AI NPCs for VR and Meta Quest - Tools Compared

A player in a Quest headset walks up to an AI NPC running a market stall, looks at a brass key on the counter and says, "How much for that one?" The shopkeeper glances at the key, names a price, and when the player says "fine, hand it over," slides it across.

No menu, no typing, no dialogue wheel. That is why VR is one of the best places for AI NPCs. It is also one of the hardest: standalone headsets have limited compute, the microphone is always close to the player's mouth, and every half-second of silence feels longer when someone is standing in front of you.

Here is what a VR character needs, how the main AI NPC tools for VR and Meta Quest compare, and how to build one.

What a VR character needs

• Voice first. Players speak and expect speech back. That means speech-to-text, a character that decides what to say, and streamed text-to-speech, so the first words play while the rest of the reply is still being generated.
• Scene awareness. In the scene above, the shopkeeper had to know the key existed and that the player meant it. Characters should react to what is near them and what just happened.
• Actions, not only words. "Hand it over" has to become something your game does.
• Embodiment. An emotion signal on each line, so you can drive facial animation, gestures and posture.
• Comfort. Short replies, clear turn-taking, and a visible "listening" and "thinking" state, because silence in VR reads as a bug.
• Portability. Quest today, PC VR or another headset tomorrow.

Can a Quest run the model itself?

Partly. Standalone headsets can run only small language models, and doing so competes with rendering for compute, heat and battery.

Meta's AI Building Blocks pair a Unity agent with a provider that runs in the cloud, on a local machine or on the headset. Recent versions run small models such as SmolLM, Qwen and Phi directly on Quest 3 and 3S, while speech-to-text and text-to-speech go to cloud providers such as OpenAI and ElevenLabs. Meta recommends Unity 6 or newer.

LLMUnity runs GGUF models through llama.cpp on PC, mobile and VR, and its docs warn that larger models typically will not work on mobile hardware.

On-device inference keeps data on the headset, but character quality and frame rate are the trade-offs. Most shipped VR characters will use cloud inference for conversation, with local models for prototyping or simple interactions.

AI NPC tools for VR and Meta Quest compared

Build a voiced Quest NPC with Neocortex

1. Design the character. In the browser editor, fill in the Agent, Personality, Emotion and Knowledge nodes, and clone a voice from a short sample you have the rights to or design one from a text description.
2. Talk to it before you build. Test the character in the browser. In-editor testing never uses credits, so tune it until it sounds right.
3. Set up your XR project. Use your usual Quest setup in Unity, for example the XR Interaction Toolkit with the Android build target.
4. Import the Unity SDK. Start from the voice sample scene, and start recording when the player presses a controller button or makes a talk gesture.
5. Make the world visible. Tag objects as interactables and send recent events, so the character knows the brass key is there and the player just picked it up.
6. Wire up actions and emotion. Replies carry an emotion tag and actions with a target. Map emotions to blend shapes and gestures, and actions to your game code.
7. Test on the headset. Check response time on the Wi-Fi your players actually use, not only on your office network.

Keeping replies fast in a headset

• Stream audio. Play the first sentence as soon as it arrives.
• Fill the gap. A nod, a "hmm" animation or a glance at the object being discussed covers the moment before speech starts.
• Keep lines short. Shape the character's personality and instructions toward short, spoken replies. Long monologues feel worse in VR than on a screen.
• Use clear turn-taking. Push-to-talk on a controller button keeps game audio and the player's side comments out of the transcript.
• Send only what matters. Pass nearby objects and recent events, not your whole scene.

Designing conversations that feel right in VR

Fast replies are half the job. The other half is how the conversation sits in the space.

Put the voice in the character. Play speech from a spatial audio source at the character's head, so the player hears who is talking and where they are.

Respect personal space. Players lean in to talk. Keep the character's idle animation calm at close range, and avoid sudden movement toward the player while it speaks.

Show that it heard you. A head turn toward the player when recording starts, and a small gesture when the transcript arrives, tell the player their words landed before the reply begins.

Offer subtitles. A small world-space caption near the character helps in noisy rooms, for players with hearing loss, and when the player is not sure what they just heard.

Let the player walk away. End or pause the conversation when the player leaves a radius, and pick it up again when they return. The character should not keep talking to an empty room.

Make objects part of the conversation. If the player points at or holds something, send it as context. "What is this?" is the most natural question in VR, and it only works if the character knows what "this" is.

Related comparisons

• Meta Quest AI Building Blocks vs Neocortex
• Convai alternative
• LLMUnity vs Neocortex

Start with one character in your headset

Neocortex characters notice the scene through interactables, carry an emotion on every line, speak in a cloned or designed voice with streamed replies, and work across engines and headsets. Several characters can even share one conversation: group chat supports up to six on the Pro and Team plans.

See our VR demo with Unity's XR Interaction Toolkit, then create a free Neocortex account and bring a character into your headset.

Tool details were last checked in October 2026.

Frequently Asked Questions