How to get audio from my Persee? Part 2.

Posted on October 2018

This article attempts to find a conclusion to the mystery of lacking audio input support for Orbbec + Unity3D. Previously it got clear that at least Orbbec Astra Pro supports ordinary audio input interface of the device, e.g. Windows as I tested it. Now, let's try to get some sense on how to get audio input working on both Astra Pro and Persee using Orbbec Astra SDK 2.0.9 Beta3d for Unity 5.5.4.

Assuming that a common audio interface can be accessed, it would be rather easy to write a standard microphone "parrot" program that would play back the audio signals recorded. Here is my test program:

using UnityEngine;
using UnityEngine.UI;

[RequireComponent(typeof(AudioSource))]
public class Parrot : MonoBehaviour
{
    public Text debugText;
    private AudioSource audioSource;

// Use this for initialization
    void Start()
    {
        audioSource = this.GetComponent<AudioSource>();

PlayAudio();
    }

// Update is called once per frame
    void Update()
    {
        float[] data = new float[256];
        float sum = 0f;

audioSource.GetOutputData(data, 0);

for (int i = 0; i < data.Length; i++)
        {
            sum += data[i];
        }

if (sum > 1)
        {
            ShowMessage("DATA SUM: " + sum);
        }
    }

private void ShowMessage(string msg)
    {
        if (debugText)
        {
            debugText.text += "\r\n" + msg;
        }
        Debug.Log(msg);
    }

private void PlayAudio()
    {
        int min = 0;
        int max = 0;
        int index = 0;
        long waitCalls = 0;

ShowMessage("Starting audio from device: " + Microphone.devices[index]);

Microphone.GetDeviceCaps(Microphone.devices[index], out min, out max);

ShowMessage("Freq: " + max);

audioSource.clip = Microphone.Start(Microphone.devices[index], true, 2, max);

while (!(Microphone.GetPosition(Microphone.devices[index]) > 1))
        {
            waitCalls++;
        }

audioSource.loop = true;
        audioSource.Play();
                
        ShowMessage("Playback started after " + waitCalls + " wait calls");
    }
}

You can bind it to an empty game object that has an empty AudioSource component in it. Like this. Also add some text object for displaying it on the screen which is useful for running it on Persee.

Basically, what it does is just display device information, play back the audio from the microphone and show the sum of audio signal values on the screen. I added this just to validate if there is any audio input from the device at all. I tested this on my laptop without plugging in any Orbbec sensors, and it shows some values when speaking or making any other noise around the microphone.

Now, let's plugin Astra Pro and run the program.

It looks all good. The program shows that microphone device called "Microphone (Orbbec audio device)" with maximum frequency of 48000 Hz is available. Once I speak something, the data sum exceeds the set threshold of 1 and shows on the screen.

Ok. Then: Persee! Compile the program for Android (see tips in the Getting Started article), install on Persee and open it.

Now, here's the surprise. The device available is named "Android audio device", not Orbbec at all. If you debug what else is available in the device list in Unity, there is nothing else, unfortunately. Frequency is 44100 Hz, the same as on my laptop. The data sum never exceed the threshold and after debugging it shows 0 as all values in the array.

So, this didn't work. At all!

It seems I cannot get audio from my Persee. I can now kiss goodbye to all hopes of using Persee for both gesture and speech recognition like the good old Kinect. The only information Orbbec is sharing about this topic is available in this forum thread. Which is no information. But maybe some day...