API Reference

After setting up the Neocortex SDK in your Unity project, you can start using the APIs to interact with the Neocortex project.

Neocortex Smart Agent Component

The Neocortex Smart Agent component is the main component that allows you to interact with the Neocortex project.

Methods

TextToText

Send a text message to the Neocortex project, and expect a text response.

Parameters:

message: The text message to send.

TextToText Example

var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnChatResponseReceived.AddListener((response) =>
{
    Debug.Log($"Message: {response.message}");
    Debug.Log($"Action: {response.action}");
    Debug.Log($"Emotion: {response.emotion.ToString()}");
});
smartAgent.TextToText("Hello, Neocortex!");

TextToAudio

Send a text message to the Neocortex project, and expect a audio response.

Parameters:

message: The text message to send.

TextToAudio Example

var audioSource = GetComponent<AudioSource>();
var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnChatResponseReceived.AddListener((response) =>
{
    Debug.Log($"Message: {response.message}");
    Debug.Log($"Action: {response.action}");
    Debug.Log($"Emotion: {response.emotion.ToString()}");
});
smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
{
    audioSource.clip = audioClip;
    audioSource.Play();
});

smartAgent.TextToAudio("Hello, Neocortex!");

AudioToText

Sends an audio clip to the Neocortex project. This method is used with NeocortexAudioReceiver component to send audio data.

Parameters:

audioClip: The audio clip to send.

AudioToText Example

var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnTranscriptionReceived.AddListener((message) =>
{
    Debug.Log($"You: {message}");
});

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
{
    Debug.Log($"Audio Data Length: {audioClip.samples}");
    smartAgent.AudioToText(audioClip);
});

// Start recording audio for 3 seconds
audioReceiver.StartMicrophone();
await Task.Delay(3000);
audioReceiver.StopMicrophone();

AudioToAudio

Sends an audio clip to the Neocortex project and expects an audio response. This method is used with NeocortexAudioReceiver component to send audio data.

Parameters:

audioClip: The audio clip to send.

AudioToAudio Example

var audioSource = GetComponent<AudioSource>();
var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
{
    audioSource.clip = audioClip;
    audioSource.Play();
});
smartAgent.OnTranscriptionReceived.AddListener((message) =>
{
    Debug.Log($"You: {message}");
});
smartAgent.OnChatResponseReceived.AddListener((response) =>
{
    Debug.Log($"Message: {response.message}");
    Debug.Log($"Action: {response.action}");
    Debug.Log($"Emotion: {response.emotion.ToString()}");
});

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
{
    Debug.Log($"Audio Data Length: {audioClip.samples}");
    smartAgent.AudioToAudio(audioClip);
});

// Start recording audio for 3 seconds
audioReceiver.StartMicrophone();
await Task.Delay(3000);
audioReceiver.StopMicrophone();

GetChatHistory

Returns the last chat messages with given limit if there was a chat session on user device.

Parameters:

limit: The number of last messages to fetch.

GetChatHistory Example

var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnChatHistoryReceived.AddListener((messages) =>
{
    foreach(var message in messages){
      Debug.Log($"{message.sender}: {message.content}");
    }
});
smartAgent.GetChatHistory(5);

Returns the string Chat Session ID generated for the current or previous chat. This string is saved in PlayerPrefs with neocortex-session-id key. If there was no previous chat session, or the session id is cleaned it will return empty string.

GetSessionID Example

var smartAgent = GetComponent<NeocortexSmartAgent>();
var sessionId = smartAgent.GetSessionID();

CleanSessionID

Cleans the session id which essentially helps start a new chat with no previous history.

CleanSessionID Example

var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.CleanSessionID();

Events

OnChatResponseReceived

Event that is triggered when the Neocortex project responds to a text message.

Arguements:

response: The response from the Neocortex project.

OnChatResponseReceived Example

var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnChatResponseReceived += (response) =>
{
    Debug.Log($"Message: {response.message}");
    Debug.Log($"Action: {response.action}");
    Debug.Log($"Emotion: {response.emotion.ToString()}");
};

OnTranscriptionReceived

Event that is triggered when the Neocortex project transcribes an audio message to text.

Arguements:

message: The transcribed audio message.

OnTranscriptionReceived Example

var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnTranscriptionReceived += (message) =>
{
    Debug.Log($"You: {message}");
};

OnAudioResponseReceived

Event that is triggered when the Neocortex project responds with an audio message.

Arguements:

audioClip: The audio clip received from the Neocortex project.

OnAudioResponseReceived Example

var audioSource = GetComponent<AudioSource>();
var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnAudioResponseReceived += (audioClip) =>
{
    audioSource.clip = audioClip;
    audioSource.Play();
};

OnChatHistoryReceived

Event that is triggered when a chat history of the current or previous session is received.

Arguements:

messages: An array of past messages.

OnChatHistoryReceived Example

var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnChatHistoryReceived += (messages) =>
{
    foreach(var message in messages){
      Debug.Log($"{message.sender}: {message.content}");
    }
};

OnRequestFailed

Event that is triggered when a request to the Neocortex project fails.

Arguements:

error: The error message.

OnRequestFailed Example

var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.OnRequestFailed += (error) =>
{
    Debug.LogError(error);
};

Neocortex Audio Receiver Component

The NeocortexAudioReceiver component is used to record audio data from the microphone via loudness of the souned, so you can have a hands free chat with the smart agent. On this component you can:

pick the microphone device to use
set the amplitude threshold for when to start and stop recording
set the max wait time for the recording to automatically stop if no sound is detected

Methods

StartMicrophone

Starts recording audio from the microphone.

StartMicrophone Example

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StartMicrophone();

StopMicrophone

Stops recording audio from the microphone

StopMicrophone Example

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StopMicrophone();

Events

OnAudioRecorded

Event that is triggered when audio data is recorded from the microphone.

Arguements:

audioClip: The recorded audio clip.

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
{
    Debug.Log($"Audio Data Length: {audioClip.samples}");
});

Neocortex Web Audio Receiver Component

The NeocortexWebAudioReceiver component is used to receive audio data from the browser in WebGL builds. This component is used in conjunction with the NeocortexSmartAgent component to send audio data to the Neocortex project.

pick the microphone device to use
set the amplitude threshold for when to start and stop recording
set the max wait time for the recording to automatically stop if no sound is detected

Methods

StartMicrophone

Starts recording audio from the microphone.

StartMicrophone Example

var audioReceiver = GetComponent<NeocortexWebAudioReceiver>();
audioReceiver.StartMicrophone();

StopMicrophone

Stops recording audio from the microphone

StopMicrophone Example

var audioReceiver = GetComponent<NeocortexWebAudioReceiver>();
audioReceiver.StopMicrophone();

Events

OnAudioRecorded

Event that is triggered when audio data is recorded from the microphone.

Arguements:

audioClip: The recorded audio clip.

var audioReceiver = GetComponent<NeocortexWebAudioReceiver>();
audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
{
    Debug.Log($"Audio Data Length: {audioClip.samples}");
});

On this page