API Reference

After setting up the Neocortex SDK in your Unity project, you can start using the APIs to interact with the Neocortex project.

Neocortex Smart Agent Component

The Neocortex Smart Agent component is the main component that allows you to interact with the Neocortex project.

Methods

TextToText

Send a text message to the Neocortex project, and expect a text response.

• Parameters:
  • message: The text message to send.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatResponseReceived.AddListener((response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
  });
  smartAgent.TextToText("Hello, Neocortex!");

TextToAudio

Send a text message to the Neocortex project, and expect a audio response.

• Parameters:
  • message: The text message to send.

  var audioSource = GetComponent<AudioSource>();
  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatResponseReceived.AddListener((response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
  });
  smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
  {
      audioSource.clip = audioClip;
      audioSource.Play();
  });
   
  smartAgent.TextToAudio("Hello, Neocortex!");

AudioToText

Sends an audio clip to the Neocortex project. This method is used with NeocortexAudioReceiver component to send audio data.

• Parameters:
  • audioClip: The audio clip to send.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnTranscriptionReceived.AddListener((message) =>
  {
      Debug.Log($"You: {message}");
  });
   
  var audioReceiver = GetComponent<NeocortexAudioReceiver>();
  audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
  {
      Debug.Log($"Audio Data Length: {audioClip.samples}");
      smartAgent.AudioToText(audioClip);
  });
   
  // Start recording audio for 3 seconds
  audioReceiver.StartMicrophone();
  await Task.Delay(3000);
  audioReceiver.StopMicrophone();

AudioToAudio

Sends an audio clip to the Neocortex project and expects an audio response. This method is used with NeocortexAudioReceiver component to send audio data.

• Parameters:
  • audioClip: The audio clip to send.

  var audioSource = GetComponent<AudioSource>();
  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
  {
      audioSource.clip = audioClip;
      audioSource.Play();
  });
  smartAgent.OnTranscriptionReceived.AddListener((message) =>
  {
      Debug.Log($"You: {message}");
  });
  smartAgent.OnChatResponseReceived.AddListener((response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
  });
   
  var audioReceiver = GetComponent<NeocortexAudioReceiver>();
  audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
  {
      Debug.Log($"Audio Data Length: {audioClip.samples}");
      smartAgent.AudioToAudio(audioClip);
  });
   
  // Start recording audio for 3 seconds
  audioReceiver.StartMicrophone();
  await Task.Delay(3000);
  audioReceiver.StopMicrophone();

Events

OnChatResponseReceived

Event that is triggered when the Neocortex project responds to a text message.

• Arguements:
  • response: The response from the Neocortex project.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatResponseReceived += (response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
  };

OnTranscriptionReceived

Event that is triggered when the Neocortex project transcribes an audio message to text.

• Arguements:
  • message: The transcribed audio message.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnTranscriptionReceived += (message) =>
  {
      Debug.Log($"You: {message}");
  };

OnAudioResponseReceived

Event that is triggered when the Neocortex project responds with an audio message.

• Arguements:
  • audioClip: The audio clip received from the Neocortex project.

  var audioSource = GetComponent<AudioSource>();
  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnAudioResponseReceived += (audioClip) =>
  {
      audioSource.clip = audioClip;
      audioSource.Play();
  };

OnRequestFailed

Event that is triggered when a request to the Neocortex project fails.

• Arguements:
  • error: The error message.

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnRequestFailed += (error) =>
  {
      Debug.LogError(error);
  };

Neocortex Audio Receiver Component

The NeocortexAudioReceiver component is used to record audio data from the microphone via loudness of the souned, so you can have a hands free chat with the smart agent. On this component you can:

• pick the microphone device to use
• set the amplitude threshold for when to start and stop recording
• set the max wait time for the recording to automatically stop if no sound is detected

Methods

StartMicrophone

Starts recording audio from the microphone.

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StartMicrophone();

StopMicrophone

Stops recording audio from the microphone

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StopMicrophone();

Events

OnAudioRecorded

Event that is triggered when audio data is recorded from the microphone.

• Arguements:
  • audioClip: The recorded audio clip.

  var audioReceiver = GetComponent<NeocortexAudioReceiver>();
  audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
  {
      Debug.Log($"Audio Data Length: {audioClip.samples}");
  });