In this article, as promised, we'll get knee-deep in coding Orbbec apps using Unity. Surprisingly, the samples didn't come with the standard skeleton renderer, only a simple class called "SkeletonRenderer" which displays the joint positions only. So, let's make this classic stickman alive!

I'm assuming that you already have the Unity sample project for Orbbec's Astra SDK which we wen't through in the previous article. If you're mostly keen to copy-paste, you can also just download the whole project.

The basic steps for doing that are listed here:

• Make a new copy of the "SkeletonRenderer" script
• Add LineRenderers with different materials
• Bind the LineRenderers to the new script
• Modify the script to use the LineRenderers for drawing lines between the joint positions according to the stickman's anatomy

I'm using Astra Pro and Windows 10. Now, let's get to the details.

First, to keep the original sample project intact, we just copy the original project as a new project with a new name. Here, "AstraSDK2.0.7BetaBodyTrackualizer".

Then, also copy the "SkeletonRenderer" script to another folder and rename it (here: "AdvancedSkeletonRenderer"... rename the class inside the script too). Good idea to copy the "TestScene" with a new name also.

Then to the code.

Replace the old "SkeletonRenderer" script with the new "AdvancedSkeletonRenderer" script in the Editor: Select for "SkeletonViewer", remove the old script, then add the new one.

Now we got one renderer done. Let's then make other similar renderers for the arms and legs. We'll need different renderers for displaying each arm and leg as well as the torso separately in different color. Repeat for each limb (4 times):

• Copy the previous child game object
• Rename it descriptively
• Select different color materials for left and right (here, I use blue for the right side, red for the left side)

Here you could create as fancy materials as you like... but let's not get into that this time.

Two important thing must also be done to every renderer: set the line with and number of point positions. In LineRenderer the line width does not mean the number of pixels or anything you could assume, so we should avoid using too thick lines or our stickman is going to look really chubby. Select all the 5 game objects and:

• Set "Width" to 0.03
• Set the array size of "Positions" to match with the number of joints in each limb: 5 for arms and torso, 4 for legs (there are no separate ankle points as in Kinect)

Finally, we have to bind the LineRenderers to our script. Click "SkeletonViewer" and assign all the parameters of "AdvancedSkeletonRenderer". Remember to select the "JointPrefab" as there is no handling for null value in the original "SkeletonRenderer".

What we have now should be enough for drawing different color lines between the relevant joints once we do the last part of coding.

One thing in the editor first. Since we removed the old "SkeletonRenderer", we must bind the body data input from the sensor to our new script. This is done as follows:

• Select "AstraController" game object
• Select the event handler called "New Body Frame Event (Body Frame)"
• Set the "SkeletonViewer" game object as its target

Continue to select "AdvancedSkeletonRenderer" component and its function "OnNewFrame". By doing this, we ensure that every time the sensor produces a new body frame, our script's function is invoked.

We can then move on to see what that "OnNewFrame" function does.

public void OnNewFrame(Astra.BodyFrame frame) { if (frame.Width == 0 || frame.Height == 0) { return; } if (_lastFrameIndex == frame.FrameIndex) { return; } _lastFrameIndex = frame.FrameIndex; frame.CopyBodyData(ref _bodies); UpdateSkeletonsFromBodies(_bodies); }

Lines 3-7 seem to just validate that the frame is not empty and lines 9-12 that the frame is new, i.e. the frame number changed from the previous frame. If things look good, the program will proceed to the function called "UpdateSkeletonsFromBodies" which handles all the body data related processing. Let's go to "UpdateSkeletonsFromBodies".

// Lines representing the body private Vector3[] leftArmPos = new Vector3[5]; private Vector3[] rightArmPos = new Vector3[5]; private Vector3[] leftLegsPos = new Vector3[4]; private Vector3[] rightLegsPos = new Vector3[4]; private Vector3[] torsoPos = new Vector3[5]; void UpdateSkeletonsFromBodies(Astra.Body[] bodies) { foreach (var body in bodies) { ... for (int i = 0; i < body.Joints.Length; i++) { ... // LEFT ARM if (i == (int)Astra.JointType.LeftHand) { leftArmPos[0] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.LeftWrist) { leftArmPos[1] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.LeftElbow) { leftArmPos[2] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.LeftShoulder) { leftArmPos[3] = skeletonJoint.transform.position; } // RIGHT ARM else if (i == (int)Astra.JointType.RightHand) { rightArmPos[0] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.RightWrist) { rightArmPos[1] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.RightElbow) { rightArmPos[2] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.RightShoulder) { rightArmPos[3] = skeletonJoint.transform.position; } // LEFT LEG else if (i == (int)Astra.JointType.LeftFoot) { leftLegsPos[0] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.LeftKnee) { leftLegsPos[1] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.LeftHip) { leftLegsPos[2] = skeletonJoint.transform.position; } // RIGHT LEG else if (i == (int)Astra.JointType.RightFoot) { rightLegsPos[0] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.RightKnee) { rightLegsPos[1] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.RightHip) { rightLegsPos[2] = skeletonJoint.transform.position; } // TORSO else if (i == (int)Astra.JointType.Head) { torsoPos[0] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.Neck) { torsoPos[1] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.MidSpine) { torsoPos[3] = skeletonJoint.transform.position; } // COMMON JOINTS else if (i == (int)Astra.JointType.ShoulderSpine) { leftArmPos[4] = skeletonJoint.transform.position; rightArmPos[4] = skeletonJoint.transform.position; torsoPos[2] = skeletonJoint.transform.position; } else if (i == (int)Astra.JointType.BaseSpine) { leftLegsPos[3] = skeletonJoint.transform.position; rightLegsPos[3] = skeletonJoint.transform.position; torsoPos[4] = skeletonJoint.transform.position; } } } }

It's a bit long function, but its original part only instantiated the joint prefab and set its position to where the joint locates. Simple. Here, I added:

• Vector3 arrays for connecting each joint to the simplemost bone structure (yes, exactly like in the song) for each limb and torso (lines 2-6)
• Storing of each joint's position to the correct Vector3 array in the existing "for" loop (lines 16-101... no editing needed, just added some code)

The point is that we can use each LineRenderer to draw one limb with a single line consisting of a couple of segments. For instance, the left arm starts from the hand, goes to wrist, elbow and shoulder (lines 16-32), and ends at the center of the shoulder (shoulder spine, line 92).

The same pattern can be used for all the limbs. Torso consists of remaining joints in the middle from head to butt, i.e. base spine.

The last part is obvious: we just use each LineRenderer to draw the lines along the bone structure using the "SetPositions" function.

That was just the simple case. This is nowhere near perfect piece of code until we:

• Support multiple bodies by instantiating more LineRenderers
• Make the joints smaller or hide them altogether (could be a nice multiplier to Transform.localScale)
• Add handling for lost user
• Add options for displaying the joints and/or skeleton
• Validate all the parameters given to the script
• ... and so on!

But maybe this is enough for starters. More coding stuff next month!