Voice Commands in VR: Hands-Free Control on Meta Quest

In VR the player's hands are the controller: one holds the wrench, the other holds the torch, and reaching for a wrist menu means letting go of one of them. Neocortex Voice Commands lets the player keep hold of both: they say "pause the simulation" or "switch to the gauge", and the command is recognised on the Meta Quest headset itself, offline. Add an AI instructor character, and the same voice that runs the scenario can ask why the pressure just spiked.

What it looks like

Halvard Station is an original VR training sim for maintenance technicians on a fictional offshore pumping platform. Today's scenario: a leaking flange in pump room two. The trainee is crouched under a pipe run with a flashlight in the left hand and a torque wrench in the right, seated on a bolt.

Without letting go of either, they say:

"Show me the isolation valve."

A soft outline appears around a wheel valve on the far wall. Then: "Pause the simulation and switch to the pressure gauge." The hiss of the leak stops mid-spray, and the wrench in the right hand becomes a handheld gauge. "Resume." The leak picks up again. "Call the control room", and a radio operator answers on the trainee's channel.

The reading is higher than the trainee expected. That isn't a command, it is a question, so it goes to the instructor, an AI character called Chief Engineer Odile Varga:

Trainee: Why did the pressure jump when I cracked that bolt? Odile: Because the line's still live. You skipped the bleed valve. Snug that bolt back down, bleed the line, then try again.

Odile knows the trainee skipped a step because the scenario sent it as an event. The procedure itself lives in her knowledge base, uploaded from the station's maintenance manual. Her reply also carried a highlight action on the bleed valve, so the trainee knows exactly where to look.

How Neocortex makes it work

• On the headset, offline. Speech recognition and command matching run on the Quest. Commands keep working in a training room with patchy Wi-Fi, and no audio leaves the headset.
• Hands stay on the task. Menus, tool swaps, scenario controls and calls to teammates all move to voice, so the player never has to put down what they are holding.
• Natural phrasing. "Pause", "freeze the sim" and "stop the simulation" can all trigger the same action. Trainees say what comes naturally under pressure, not a phrase from a handout.
• Several actions in one sentence. "Pause the simulation and switch to the pressure gauge" arrives as two commands.
• Context. "Show me the bleed valve. Now open it." resolves "it" to the valve just highlighted.
• Push-to-talk or always listening. A grip or face button can act as push-to-talk when one finger is free. In training where both hands are always busy, always listening often fits better.
• Command sets of actors, actions and objects, designed in the Neocortex dashboard: teammates and the control room as actors; pause, show, switch and call as actions; valves, tools and rooms as objects.
• Multiple languages. One command set per language, so crews can train in the language they work in.

Odile is a Neocortex AI character built in the visual node editor, with a personality, a streamed voice and emotion tags on each line. Her knowledge base takes PDF or TXT uploads (1 on Free, 5 on Starter, 20 on Pro, 100 on Team). Your scenario sends events such as "trainee opened the flange before bleeding the line", and she returns actions you authored, like highlighting a part, with the target your app should light up. A custom instructor voice is available on Pro (5 custom voices) and Team (20).

Ideas to build

A VR shooter squad. Rifle up with both hands, the player calls "Mara, smoke the courtyard. Jensen, take the left door." Each squadmate is an actor, so the orders split cleanly, and each can be an AI character who answers back in their own voice when the player asks for a status report.

A VR building tool. In a creative app the hands sculpt and place, and the voice runs the toolbar: "switch to the brush", "make it twice as big", "duplicate that", "undo". Nobody has to dig through a floating palette mid-stroke, and the work stays in front of them.

A VR puzzle room. Players hold a lantern in one hand and a lever in the other while they work out the room: "lower the bridge", "rotate the left mirror". Pair it with an AI character who gives hints when asked, and the room can be generous or stingy depending on how the player is doing.

Scenario control for trainers. In a longer training programme, voice lets the trainee reset and vary the drill without leaving the headset: "restart from the leak", "run it again with the alarm on". The scenario becomes something they can repeat ten times in a session instead of twice.

Building it

Neocortex Voice Commands runs on the Meta Quest headset, inside your VR app. AI characters such as the instructor connect through the Neocortex API, and our open-source VR demo is a working Quest project with AI characters to start from.

1. Find the moments when hands are full. Walk through your experience and mark every tool swap, menu, pause and call for help. Those become your commands.
2. Design a command set in the Neocortex dashboard: teammates and stations as actors, your verbs as actions, and the equipment and places in the scene as objects, with a few example phrases for each.
3. Add Neocortex Voice Commands to your Quest build and connect each matched command to your scenario controller and tool system.
4. Pick a listening mode that matches the hands: always listening for drills where both hands stay busy, push-to-talk on a grip button for games with lots of chatter.
5. Add the AI instructor: a Neocortex character with a personality, a voice, a knowledge base of your procedures and actions for highlighting equipment. Send scenario events with each question so the answers follow what the trainee actually did.
6. Add a command set per language your trainees or players speak.

Give every command a small acknowledgement in the headset, such as a soft chime or a short label near the wrist. With no screen edge to glance at, players need to know they were heard. Keep the wrist menu reachable by hand as well, for anyone who would rather reach than speak.

Voice Commands is a licensed SDK: contact us for a Voice Commands license. The instructor and other AI characters, with the API behind them, are part of every plan, Free included.

Hands-free VR is one entry in our AI NPC, ambient AI and voice command use cases. For more on headsets, read VR soft skills training with AI and AI characters for VR escape rooms, and see the VR experiences page. Start with one tool swap and one pause command, and watch how quickly trainees stop reaching for the menu.

Checked October 2026.

Frequently Asked Questions