Meta Quest AI Building Blocks vs Neocortex for VR NPCs

You are building for Meta Quest in Unity and you want a character the player can talk to. Should you start with Meta's own AI Building Blocks, or with a character platform?

Meta's tools are the obvious first stop. The Meta XR SDK includes AI Building Blocks for language models, speech and text-to-speech, and the long-standing Voice SDK handles voice commands and dictation. They are useful plumbing. They are not characters. This comparison covers what each side gives you for a VR NPC, and what is left over.

Know what Meta ships for AI on Quest

• AI Building Blocks. Four blocks: Object Detection, Large Language Models, Speech to Text and Text to Speech. Each pairs a Unity component with a provider that decides where inference runs: in the cloud, on a local machine, or on the headset. Meta's documentation names providers such as OpenAI, Ollama, Hugging Face, Replicate and ElevenLabs.
• Requirements. Unity 6 or newer, a Quest 3 or 3S, and an internet connection for cloud providers. Blocks that use the passthrough camera also need recent Meta XR Core SDK and MR Utility Kit versions.
• Voice SDK. Built on Wit.ai, for voice commands, dictation and TTS.
• A changing model backend. Meta has retired its Llama API preview, so projects that used it for the language model block need another provider.

Meta also warns that cloud models may not always be available on a provider's servers, so you check provider and model availability before you ship.

Count what you still build

Building Blocks connect models. A character needs more. On top of the blocks, your team writes and maintains:

• The character itself. Persona, backstory, knowledge, memory and guardrails, kept consistent across a long play session.
• The glue between blocks. Speech-to-text output into the language model, the reply into text-to-speech, with interruption handling so the player can cut in.
• Separate provider accounts. Each block can bill through a different company, with its own keys, limits and price changes.
• Custom voices through a third-party voice service.
• Scene awareness and actions. Telling the character what the player is holding or looking at, and turning a reply into something that happens in the world.
• Every other platform. Building Blocks target Quest in Unity. A PC VR build, a flat-screen version or an Unreal port starts over.

Compare Meta AI Building Blocks and Neocortex

Walk through a mixed reality shopkeeper

Picture a potion seller standing on the player's real desk in passthrough.

With Building Blocks, you drop in the speech-to-text, language model and text-to-speech blocks, pick three providers and wire them together. The shopkeeper's personality lives in a system prompt. To make her react when the player picks up a bottle, you write code that describes the scene in text, adds it to the prompt, parses the reply for intent, and triggers the animation. Her voice is a stock preset unless you set up a separate voice service.

With Neocortex, you build her in the editor first: an Agent node for the backstory, Personality and Emotion nodes for her temper, a Knowledge file with the price list, and an Objects node listing the bottles. You test her in the browser until she haggles the way you want, without spending credits. In Unity, your game tells her which tagged object the player is holding. She answers in her own cloned or designed voice, streamed so the reply starts quickly, and the reply carries an emotion tag for her face plus an action in the shape of { "name": "OPEN_DOOR", "targetId": "door_02" } that your code runs, for example opening the cabinet behind her. If the quest needs structure, a Conversation Flow moves her from greeting to haggling to the secret recipe, and the API reports each checkpoint as flowState.

You can see a character running in a headset in our VR demo with Unity's XR Interaction Toolkit.

Pick the right fit for your Quest project

Building Blocks fit a team that only needs a single capability, such as dictation into a text field or object detection for a mixed reality feature, and is ready to own the provider accounts.

Neocortex fits a team that wants characters: designers who edit personality and knowledge without rebuilding the app, voices that belong to each character, and NPCs that act on the scene. It also fits any project that may leave Quest, because the same character runs on other platforms and engines.

Both can live in one app. Use the Voice SDK for menu commands like "pause" or "inventory", and a Neocortex character for conversation.

On plans: you can build and test for free with 2 characters, commercial releases need Pro (EUR 29 a month) or Team (EUR 99 a month), and Pro adds group chat for up to six characters over the API, which suits a party of companions in VR.

Put one character in the headset

Start with the NPC your players will meet first. Build it in the browser, add the Unity SDK, and hear it answer in the headset. Every later character follows the same steps.

Meta details were last checked in October 2026.

Frequently Asked Questions