Building Voice NPCs on the OpenAI Realtime API vs Using Neocortex

On Friday night you connect a microphone to gpt-realtime-2.1, write a system prompt for a grumpy blacksmith, and by Sunday he is insulting your armour choices out loud. It feels finished. It is not.

Speech-to-speech models make a talking NPC demo an afternoon's work. Shipping that NPC to players takes much longer, and almost none of the extra work is about the model. This post lays out what building voice NPCs directly on the OpenAI Realtime API involves in Unity, and where Neocortex fits.

What the Realtime API gives you

• Speech-to-speech models, currently led by gpt-realtime-2.1, with configurable reasoning effort, function calling and a 128K context window.
• Connections over WebRTC for browsers and mobile, WebSocket from a server, and SIP for phone calls.
• Short-lived client secrets your server creates for each session, so the API key stays off the client.
• Built-in voices, ten of them. The voice cannot change once the model has spoken in a session.
• Custom voices from recordings, limited to eligible customers who contact OpenAI sales, with a consent recording from the voice owner.
• Token-based pricing for audio and text, input and output.
• Sessions capped at 60 minutes.
• Official SDKs for JavaScript, Python, .NET, Java, Go and Ruby. Unity support comes from community libraries.

DIY on OpenAI Realtime vs Neocortex

The eight things you would build yourself

To ship a voiced NPC on the Realtime API, your team owns:

1. A Unity audio client that captures the microphone, streams it, and plays the reply without gaps, including in WebGL builds, where Unity has no built-in microphone input.
2. A small backend that authenticates players and mints a client secret per session, because your API key can never ship in a game build.
3. Character design as prompts, kept consistent across every model update and every writer who touches them.
4. Session lifecycle: reconnecting before the 60-minute cap, and carrying the conversation across the gap.
5. Game integration, so function calls turn into actions on the right objects in your scene.
6. Guardrails, so the blacksmith stays a blacksmith when players try to make him something else.
7. Cost control, since every second of audio in and out is billed in tokens.
8. Upkeep every time models, prices or APIs change, across every engine you ship on.

None of it is impossible. All of it is time you are not spending on your game.

A shopkeeper NPC, built both ways

Say the player walks into a shop and asks the owner for a healing potion. The owner should answer in character, sound annoyed if the player was rude earlier, and actually hand over the item.

On the Realtime API

You write a prompt describing the shopkeeper and the shop's stock, and define a give_item function. Your backend mints a client secret, your Unity client opens the session, streams the player's voice and plays the reply. When the model calls give_item, your code finds the potion and moves it to the inventory. For the shopkeeper's mood, you track rudeness in game state and inject it into the prompt, then add an animation cue by parsing the text or asking the model for structured output. If the voice should be unique to this character, you check whether you qualify for custom voices.

On Neocortex

You build the shopkeeper in the visual editor: Personality and a starting Emotion for temperament, Knowledge for the stock list, an Intent for buying and an Action for handing over an item. You clone or design a voice for him. In Unity, your game sends nearby objects and recent events, including the player's rude remark. The reply arrives as lines with an emotion tag, so the animator knows he is annoyed, plus an action like { "name": "GIVE_ITEM", "targetId": "potion_01" } that your game executes. You test the whole exchange in the browser first, without using credits.

Both versions use a capable model. The difference is where the shopkeeper lives: in a prompt string and glue code your engineers maintain, or in a character a designer can open, read and change without touching the build.

Build the game, not the plumbing

The real cost of the DIY route shows up after launch. A model update changes how the blacksmith phrases things, a price change shifts your margins, a new Unity version breaks the community library you relied on. Each one lands on your team, usually while you are busy with something else.

The Realtime API is a model you build a voice stack around. Neocortex is the voice stack, with the character, the engine SDKs and the gameplay hooks already in place, maintained by us as models and APIs change. Designers edit personality, emotion, knowledge, intents and actions as nodes in a graph rather than a prompt buried in code, and your engineers spend their weeks on the game.

OpenAI details were last checked in October 2026.

Frequently Asked Questions