How to Add Voice Chat to AI NPCs in Unity

Typing to a character is fine. Speaking to one and hearing an answer in its own voice is a different game. This guide adds full voice chat to an AI NPC in Unity with the Neocortex SDK: the player's microphone in, a spoken reply out, plus lip sync and the platform details that usually take a week to find.

How voice chat works in the SDK

One voice turn passes through three stages: speech-to-text hears the player, the character writes a reply, and text-to-speech says it. Our AI NPC voice pipeline guide explains each one. In Unity you don't wire them yourself: the SDK handles the whole chain from three components on one GameObject.

If you haven't set up the SDK yet, start with how to add an AI character to your Unity game.

Step 1: the no-code voice NPC

Add the three components to your character's GameObject and pick the character in the Smart Agent's dropdown. With Auto Voice Input on (the default), the agent finds the receiver, sends each recording as a voice message and opens the microphone again after every reply. Press Play and talk.

The quickest route is right-click in the Hierarchy > Neocortex > Complete Voice Chat, which builds the same rig with a chat panel and a microphone button.

Step 2: choose voice detection or push-to-talk

The Audio Receiver listens with voice detection by default. Three settings shape it:

• Amplitude Threshold (default 0.1): how loud the player must be to count as speaking. Raise it in noisy rooms.
• Max Wait Time (default 1 second): how much silence ends a sentence. Shorter feels faster, but cuts off slow speakers.
• Lead In (default 0.3 seconds): audio kept from just before the player started, so first syllables aren't clipped.

For push-to-talk, switch the receiver over and start and stop it from your own input. This uses Unity's Input System:

using UnityEngine;
using UnityEngine.InputSystem;
using Neocortex;

public class PushToTalk : MonoBehaviour
{
    [SerializeField] private NeocortexAudioReceiver microphone;

    private void Start() => microphone.SetPushToTalk(true);

    private void Update()
    {
        var keyboard = Keyboard.current;
        if (keyboard == null) return;
        if (keyboard.vKey.wasPressedThisFrame) microphone.StartMicrophone();
        if (keyboard.vKey.wasReleasedThisFrame) microphone.StopMicrophone();
    }
}

Releasing V ends the recording, and the agent sends it straight away. For an on-screen button, the SDK's Neocortex Push To Talk Button does the same on press and release, and the Neocortex Microphone Dropdown lets players pick their device.

Step 3: shape the spoken reply

The Smart Agent's Chat Lines Mode decides how the reply is voiced:

• PerLineAudio voices each line as it starts, so a long reply begins speaking quickly.
• SingleAudio voices the whole reply as one clip.
• Off hands you the clip through OnAudioResponseReceived and plays nothing, for your own audio system.

Speech streams from our servers and starts playing after a fraction of a second of buffer, so the character starts talking before the whole clip exists. The voice follows each line's emotion, so an alarmed line sounds alarmed.

Two events help with presentation: OnTranscriptionReceived gives you what the player said, for subtitles or a "you said" line, and OnChatLineStarted gives you each line of the reply as it begins.

Step 4: add lip sync

Add Neocortex Face Animator (Add Component > Neocortex) to the character. It finds the agent and the audio source, and moves the mouth from the spoken text using ARKit-named blend shapes such as JawOpen and MouthFunnel. Ready Player Me avatars have these out of the box. Mapping emotions to facial expressions is yours to set: our guide to emotional AI NPCs has a practical table.

Platform notes

• Android and iOS: the receiver requests microphone permission itself on first use. Its On Permission Denied event lets you show a message or fall back to text.
• WebGL: Unity's microphone doesn't exist in browsers. Select the Neocortex WebGL template under Player Settings > Resolution and Presentation (the SDK imports it into your project, and Tools > Neocortex > Import WebGL Template restores it). Host the build over HTTPS, since browsers only allow the microphone on secure pages. If the template isn't selected, the console warns that microphone input won't work.
• Desktop: players may need to allow microphone access in their operating system's privacy settings the first time.

Common problems

• The NPC answers its own voice. On speakers, the microphone can hear the reply. Voice detection reopens after the reply ends, but push-to-talk or headphones avoid it entirely.
• Players get cut off mid-sentence. Raise Max Wait Time a little, or switch to push-to-talk.
• Short clicks trigger replies. Sounds under 0.15 seconds are already ignored. Raise Amplitude Threshold if background noise still gets through.
• Nothing happens in a web build. Check the WebGL template and HTTPS first.

Next steps

Voice is where latency shows most, so read how to get AI NPC voice replies under a second. To voice your scripted lines in the same voice, see text-to-speech for NPC dialogue in Unity, and for headsets see AI characters in a Meta Quest game. All tutorials are in the guide to building AI NPCs.

Frequently Asked Questions