Building Voice NPCs on the OpenAI Realtime API vs Using Neocortex
With speech-to-speech models like OpenAI's gpt-realtime, a talking NPC demo takes an afternoon. Shipping one takes much longer.
This post walks through what building voice NPCs directly on the OpenAI Realtime API involves in Unity, and where Neocortex fits.
What the Realtime API gives you
From OpenAI's docs (checked October 2026):
- Speech-to-speech models with function calling, image input and a 128K context window.
- Connections over WebRTC for browsers and mobile, or WebSocket from a trusted server.
- Token-based pricing for audio and text input and output, with no fixed per-minute rate.
- Sessions capped at 60 minutes.
- No official Unity SDK. Only community ports exist.
What you build yourself
To ship a voiced NPC on the Realtime API, your team owns:
- A Unity audio client for capturing the microphone, streaming it and playing the reply, including WebGL.
- A backend that issues short-lived keys, because your API key can never ship in a game build.
- Character design as prompts, kept consistent across every update.
- Memory across sessions, and reconnecting before the 60-minute limit.
- Game integration, so function calls turn into actions in your scene.
- Guardrails, so characters stay in character and on brand.
- Cost control, since every second of audio is billed in tokens.
- Upkeep every time models, prices or APIs change.
None of it is impossible. All of it is time you are not spending on your game.
DIY on OpenAI vs Neocortex at a glance
| DIY on OpenAI Realtime | Neocortex | |
|---|---|---|
| Unity client | Write your own | Unity SDK with voice samples |
| Backend for keys | Write your own | Included |
| Character design | Prompts in code | Visual node editor |
| Voice | Preset model voices | Voice cloning and voice design |
| Game actions | Wire up function calls | Actions and interactable objects built in |
| Engines | Whatever you build | Unity, Unreal, Godot (beta), Web API |
| Model choice | OpenAI only | Managed for you, plus local LLMs in Unity via Ollama |
| Cost | Tokens per second of audio | Monthly plans with included credits |
Why teams choose Neocortex
The demo is the easy part; we handle the rest. Character consistency, voice streaming, session management, usage metering and SDK upkeep across engines are what Neocortex maintains, so you do not have to.
Designers stay in control. Personality, emotion, knowledge, intents and actions are nodes in a visual graph, not a prompt buried in code.
Not locked to one model provider. Prototype with open models running locally through Ollama in Unity, and ship through the cloud.
When DIY is the right call
If AI conversation is your core technology and you have engineers to build and maintain the stack, going direct gives you maximum control. If AI characters are a feature of your game rather than the product itself, Neocortex gets you there in hours instead of weeks.
OpenAI details were last checked in October 2026.
Written by
Sercan Altundas
Date
Oct 17, 2026
Read time
4 min
