API Reference

After setting up the Neocortex SDK in your Unity project, you can start using the APIs to interact with the Neocortex project.

Neocortex Smart Agent Component

The Neocortex Smart Agent component is the main component that allows you to interact with the Neocortex project.

Methods

TextToText

Send a text message to the Neocortex project, and expect a text response.

• Parameters:
  • message: The text message to send.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatResponseReceived.AddListener((response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
      Debug.Log($"Emotion: {response.emotion.ToString()}");
  });
  smartAgent.TextToText("Hello, Neocortex!");

TextToAudio

Send a text message to the Neocortex project, and expect a audio response.

• Parameters:
  • message: The text message to send.

  var audioSource = GetComponent<AudioSource>();
  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatResponseReceived.AddListener((response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
      Debug.Log($"Emotion: {response.emotion.ToString()}");
  });
  smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
  {
      audioSource.clip = audioClip;
      audioSource.Play();
  });
   
  smartAgent.TextToAudio("Hello, Neocortex!");

AudioToText

Sends an audio clip to the Neocortex project. This method is used with NeocortexAudioReceiver component to send audio data.

• Parameters:
  • audioClip: The audio clip to send.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnTranscriptionReceived.AddListener((message) =>
  {
      Debug.Log($"You: {message}");
  });
   
  var audioReceiver = GetComponent<NeocortexAudioReceiver>();
  audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
  {
      Debug.Log($"Audio Data Length: {audioClip.samples}");
      smartAgent.AudioToText(audioClip);
  });
   
  // Start recording audio for 3 seconds
  audioReceiver.StartMicrophone();
  await Task.Delay(3000);
  audioReceiver.StopMicrophone();

AudioToAudio

Sends an audio clip to the Neocortex project and expects an audio response. This method is used with NeocortexAudioReceiver component to send audio data.

• Parameters:
  • audioClip: The audio clip to send.

  var audioSource = GetComponent<AudioSource>();
  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
  {
      audioSource.clip = audioClip;
      audioSource.Play();
  });
  smartAgent.OnTranscriptionReceived.AddListener((message) =>
  {
      Debug.Log($"You: {message}");
  });
  smartAgent.OnChatResponseReceived.AddListener((response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
      Debug.Log($"Emotion: {response.emotion.ToString()}");
  });
   
  var audioReceiver = GetComponent<NeocortexAudioReceiver>();
  audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
  {
      Debug.Log($"Audio Data Length: {audioClip.samples}");
      smartAgent.AudioToAudio(audioClip);
  });
   
  // Start recording audio for 3 seconds
  audioReceiver.StartMicrophone();
  await Task.Delay(3000);
  audioReceiver.StopMicrophone();

GetChatHistory

Returns the last chat messages with given limit if there was a chat session on user device.

• Parameters:
  • limit: The number of last messages to fetch.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatHistoryReceived.AddListener((messages) =>
  {
      foreach(var message in messages){
        Debug.Log($"{message.sender}: {message.content}");
      }
  });
  smartAgent.GetChatHistory(5);

GetSessionID

Returns the string Chat Session ID generated for the current or previous chat. This string is saved in PlayerPrefs with neocortex-session-id key. If there was no previous chat session, or the session id is cleaned it will return empty string.

var smartAgent = GetComponent<NeocortexSmartAgent>();
var sessionId = smartAgent.GetSessionID();

CleanSessionID

Cleans the session id which essentially helps start a new chat with no previous history.

var smartAgent = GetComponent<NeocortexSmartAgent>();
smartAgent.CleanSessionID();

Events

OnChatResponseReceived

Event that is triggered when the Neocortex project responds to a text message.

• Arguements:
  • response: The response from the Neocortex project.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatResponseReceived += (response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
      Debug.Log($"Emotion: {response.emotion.ToString()}");
  };

OnTranscriptionReceived

Event that is triggered when the Neocortex project transcribes an audio message to text.

• Arguements:
  • message: The transcribed audio message.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnTranscriptionReceived += (message) =>
  {
      Debug.Log($"You: {message}");
  };

OnAudioResponseReceived

Event that is triggered when the Neocortex project responds with an audio message.

• Arguements:
  • audioClip: The audio clip received from the Neocortex project.

  var audioSource = GetComponent<AudioSource>();
  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnAudioResponseReceived += (audioClip) =>
  {
      audioSource.clip = audioClip;
      audioSource.Play();
  };

OnChatHistoryReceived

Event that is triggered when a chat history of the current or previous session is received.

• Arguements:
  • messages: An array of past messages.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatHistoryReceived += (messages) =>
  {
      foreach(var message in messages){
        Debug.Log($"{message.sender}: {message.content}");
      }
  };

OnRequestFailed

Event that is triggered when a request to the Neocortex project fails.

• Arguements:
  • error: The error message.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnRequestFailed += (error) =>
  {
      Debug.LogError(error);
  };

Neocortex Audio Receiver Component

The NeocortexAudioReceiver component is used to record audio data from the microphone via loudness of the souned, so you can have a hands free chat with the smart agent. On this component you can:

• pick the microphone device to use
• set the amplitude threshold for when to start and stop recording
• set the max wait time for the recording to automatically stop if no sound is detected

Methods

StartMicrophone

Starts recording audio from the microphone.

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StartMicrophone();

StopMicrophone

Stops recording audio from the microphone

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StopMicrophone();

Events

OnAudioRecorded

Event that is triggered when audio data is recorded from the microphone.

• Arguements:
  • audioClip: The recorded audio clip.

  var audioReceiver = GetComponent<NeocortexAudioReceiver>();
  audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
  {
      Debug.Log($"Audio Data Length: {audioClip.samples}");
  });

Neocortex Web Audio Receiver Component

The NeocortexWebAudioReceiver component is used to receive audio data from the browser in WebGL builds. This component is used in conjunction with the NeocortexSmartAgent component to send audio data to the Neocortex project.

• pick the microphone device to use
• set the amplitude threshold for when to start and stop recording
• set the max wait time for the recording to automatically stop if no sound is detected

Methods

StartMicrophone

Starts recording audio from the microphone.

var audioReceiver = GetComponent<NeocortexWebAudioReceiver>();
audioReceiver.StartMicrophone();

StopMicrophone

Stops recording audio from the microphone

var audioReceiver = GetComponent<NeocortexWebAudioReceiver>();
audioReceiver.StopMicrophone();

Events

OnAudioRecorded

Event that is triggered when audio data is recorded from the microphone.

• Arguements:
  • audioClip: The recorded audio clip.

  var audioReceiver = GetComponent<NeocortexWebAudioReceiver>();
  audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
  {
      Debug.Log($"Audio Data Length: {audioClip.samples}");
  });