Neocortex
IntegrationsUnity SDK

API Reference

Learn about the NeoCortex Unity SDK methods and events

After setting up the Neocortex SDK in your Unity project, you can start using the APIs to interact with the Neocortex project.

Neocortex Smart Agent Component

The Neocortex Smart Agent component is the main component that allows you to interact with the Neocortex project.

Methods

TextToText

Send a text message to the Neocortex project, and expect a text response.

  • Parameters:
    • message: The text message to send.
    TextToText Example
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnChatResponseReceived.AddListener((response) =>
    {
        Debug.Log($"Message: {response.message}");
        Debug.Log($"Action: {response.action}");
    });
    smartAgent.TextToText("Hello, Neocortex!");

TextToAudio

Send a text message to the Neocortex project, and expect a audio response.

  • Parameters:
    • message: The text message to send.
    TextToAudio Example
    var audioSource = GetComponent<AudioSource>();
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnChatResponseReceived.AddListener((response) =>
    {
        Debug.Log($"Message: {response.message}");
        Debug.Log($"Action: {response.action}");
    });
    smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
    {
        audioSource.clip = audioClip;
        audioSource.Play();
    });
     
    smartAgent.TextToAudio("Hello, Neocortex!");

AudioToText

Sends an audio clip to the Neocortex project. This method is used with NeocortexAudioReceiver component to send audio data.

  • Parameters:
    • audioClip: The audio clip to send.
    AudioToText Example
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnTranscriptionReceived.AddListener((message) =>
    {
        Debug.Log($"You: {message}");
    });
     
    var audioReceiver = GetComponent<NeocortexAudioReceiver>();
    audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
    {
        Debug.Log($"Audio Data Length: {audioClip.samples}");
        smartAgent.AudioToText(audioClip);
    });
     
    // Start recording audio for 3 seconds
    audioReceiver.StartMicrophone();
    await Task.Delay(3000);
    audioReceiver.StopMicrophone();

AudioToAudio

Sends an audio clip to the Neocortex project and expects an audio response. This method is used with NeocortexAudioReceiver component to send audio data.

  • Parameters:
    • audioClip: The audio clip to send.
    AudioToAudio Example
    var audioSource = GetComponent<AudioSource>();
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
    {
        audioSource.clip = audioClip;
        audioSource.Play();
    });
    smartAgent.OnTranscriptionReceived.AddListener((message) =>
    {
        Debug.Log($"You: {message}");
    });
    smartAgent.OnChatResponseReceived.AddListener((response) =>
    {
        Debug.Log($"Message: {response.message}");
        Debug.Log($"Action: {response.action}");
    });
     
    var audioReceiver = GetComponent<NeocortexAudioReceiver>();
    audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
    {
        Debug.Log($"Audio Data Length: {audioClip.samples}");
        smartAgent.AudioToAudio(audioClip);
    });
     
    // Start recording audio for 3 seconds
    audioReceiver.StartMicrophone();
    await Task.Delay(3000);
    audioReceiver.StopMicrophone();

GetChatHistory

Returns the last chat messages with given limit if there was a chat session on user device.

  • Parameters:
    • limit: The number of last messages to fetch.
    GetChatHistory Example
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnChatHistoryReceived.AddListener((messages) =>
    {
        foreach(var message in messages){
          Debug.Log($"{message.sender}: {message.content}");
        }
    });
    smartAgent.GetChatHistory(5);

GetSessionID

Returns the string Chat Session ID generated for the current or previous chat. This string is saved in PlayerPrefs with neocortex-session-id key. If there was no previous chat session, or the session id is cleaned it will return empty string.

GetSessionID Example
var smartAgent = GetComponent<NeocortexSmartAgent>();
var sessionId = smartAgent.GetSessionID();

CleanSessionID

Cleans the session id which essentially helps start a new chat with no previous history.

CleanSessionID Example
var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.CleanSessionID();

Events

OnChatResponseReceived

Event that is triggered when the Neocortex project responds to a text message.

  • Arguements:
    • response: The response from the Neocortex project.
    OnChatResponseReceived Example
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnChatResponseReceived += (response) =>
    {
        Debug.Log($"Message: {response.message}");
        Debug.Log($"Action: {response.action}");
    };

OnTranscriptionReceived

Event that is triggered when the Neocortex project transcribes an audio message to text.

  • Arguements:
    • message: The transcribed audio message.
    OnTranscriptionReceived Example
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnTranscriptionReceived += (message) =>
    {
        Debug.Log($"You: {message}");
    };

OnAudioResponseReceived

Event that is triggered when the Neocortex project responds with an audio message.

  • Arguements:
    • audioClip: The audio clip received from the Neocortex project.
    OnAudioResponseReceived Example
    var audioSource = GetComponent<AudioSource>();
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnAudioResponseReceived += (audioClip) =>
    {
        audioSource.clip = audioClip;
        audioSource.Play();
    };

OnChatHistoryReceived

Event that is triggered when a chat history of the current or previous session is received.

  • Arguements:
    • messages: An array of past messages.
    OnChatHistoryReceived Example
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnChatHistoryReceived += (messages) =>
    {
        foreach(var message in messages){
          Debug.Log($"{message.sender}: {message.content}");
        }
    };

OnRequestFailed

Event that is triggered when a request to the Neocortex project fails.

  • Arguements:
    • error: The error message.
    OnRequestFailed Example
    var smartAgent = GetComponent<NeocortexSmartAgent>();
    smartAgent.OnRequestFailed += (error) =>
    {
        Debug.LogError(error);
    };

Neocortex Audio Receiver Component

The NeocortexAudioReceiver component is used to record audio data from the microphone via loudness of the souned, so you can have a hands free chat with the smart agent. On this component you can:

  • pick the microphone device to use
  • set the amplitude threshold for when to start and stop recording
  • set the max wait time for the recording to automatically stop if no sound is detected

Methods

StartMicrophone

Starts recording audio from the microphone.

StartMicrophone Example
var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StartMicrophone();

StopMicrophone

Stops recording audio from the microphone

StopMicrophone Example
var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StopMicrophone();

Events

OnAudioRecorded

Event that is triggered when audio data is recorded from the microphone.

  • Arguements:
    • audioClip: The recorded audio clip.
    var audioReceiver = GetComponent<NeocortexAudioReceiver>();
    audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
    {
        Debug.Log($"Audio Data Length: {audioClip.samples}");
    });

Neocortex Web Audio Receiver Component

The NeocortexWebAudioReceiver component is used to receive audio data from the browser in WebGL builds. This component is used in conjunction with the NeocortexSmartAgent component to send audio data to the Neocortex project.

  • pick the microphone device to use
  • set the amplitude threshold for when to start and stop recording
  • set the max wait time for the recording to automatically stop if no sound is detected

Methods

StartMicrophone

Starts recording audio from the microphone.

StartMicrophone Example
var audioReceiver = GetComponent<NeocortexWebAudioReceiver>();
audioReceiver.StartMicrophone();

StopMicrophone

Stops recording audio from the microphone

StopMicrophone Example
var audioReceiver = GetComponent<NeocortexWebAudioReceiver>();
audioReceiver.StopMicrophone();

Events

OnAudioRecorded

Event that is triggered when audio data is recorded from the microphone.

  • Arguements:
    • audioClip: The recorded audio clip.
    var audioReceiver = GetComponent<NeocortexWebAudioReceiver>();
    audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
    {
        Debug.Log($"Audio Data Length: {audioClip.samples}");
    });