Unity + Kinect Documentation

Today I finally realized the importance of writing a good documentation once you finished a tough project, as you might have to deal with the same issues next time but totally forget how you fixed it. (It happens today..again..) It’s also an opportunity for me to improve the writing skill and articulate my thoughts as I am not a native English speaker.

I was working on a VR project (actually just 360 videos). In the team of three, my role is to figure out how to connect Unity and Kinect. Our goal is to create an immersive storytelling experience where the audience could interact with the object they see in the VR scene by making the intuitive pose: wave their hands to interact with birds, squat to trigger the wave… We have bare experience in either Unity/C# or Kinect. Although there are numerous unity tutorials, the resource about connecting unity and Kinect is limited. I really had a hard time to figure this out, so hope my documentation could help people who are struggling on it.

Tools I am using

Kinect v1, win8, Unity, VR Oculus Rift

Process

First, Connect Kinect:

1. Install Kinect Windows SDK v1.7, runtime SDK v1.7, developer toolkit v1.7. as I was using win8 at that time for my Kinect 360 (not xbox), do not connect Kinect to USB 3.0 port until all the installation is done,

Kinect for Windows SDK

2. Plug in Kinect to USB 3.0 port, the green light should be turned on, flashing means it’s not working properly

3. Run the sample file “Body Skeleton” from the developer toolkit. If you cannot see the skeleton, just move back and forth (or wave your hand), change your distance to Kinect (at least 1 meter). When your skeleton shows up, it means Kinect has been connected successfully.

• *The version of SDK needed depends on the system you are using. Getting the correct version of SDK is really important, unfortunately, the documentation on the Microsoft website is really not clear, so please try those sample files to make sure you get the right one.

Second, Kinect gesture recognition in Unity:

1. Download Kinect with MS-SDK from unity assets store, import everything to the project.

I had tried all the free kinect+ unity plugin in assets store and this one works best for people who are not familiar with C# but just want to get this done in a quick and easy way. If you know how to code in C#, then it is not necessary to use this asset, as it actually could bring unexpected bugs.

2. There are 4 demos, KinectGesturesDemo is what we need.

The tutorial below talks about the basic functions of this plugin, it’s in Spanish without caption but still make sense. Quickly go through this video and you will understand how this plugin works.

The How to Use Gestures or Create Your Own Gestures.pdf inside the assets package also explains how this plugin works, including a list of Currently Recognized Gestures (i.e. swipeUp/swipeDown)

Trigger Animations:

1. Tutorial: how to trigger the animation with keypress in unity

keypress to trigger animations in Unity

2. Edit the gesture listener script to the gesture we want to recognize and then attach to the main camera

3. Edit the animation trigger script: change the condition (replace key press to gesture recognition), attach to the game object. Basically what you need to do is just adapt the gesture recognition (i.e. SwipeUp) as the condition in an if statement:

if(gestureListener.IsSwipeUp()
anim.Play (“animationControl”);)

Build Unity VR Project (Oculus)

1. Download and import Oculus Utilities for unity package. (to test VR in Unity: download oculus integration from assets store)
2. Build & setting-player-check “support Virtual Reality”
3. * project setting-player-other setting: add preload scenes/objects

4. Target platform: Windows, then Build!

5. Copy all the 3 files to the testing computer (data folder, .exe, .dll)

6. Run the .exe file, you are good to go!

Unity 360 View Tutorials

Render 360 img as skybox:

Render 360 video inside a sphere:

Render 360 image inside a sphere and keep the transparent part: create a material, attach the png image, and use this shader:

https://forum.unity.com/threads/render-only-the-inside-of-sphere-and-keep-the-transparency-as-well.520852/

Summary

Dealing with Unity and Kinect at the same time is definitely not an easy task. We encountered weird compatibility issues when trying to run Kinect on mac. Then, it’s hard to keep the audience face Kinect all the time as they cannot see the physical world with the VR headset, and they tend to go around in circles, which makes Kinect hard to detect skeletons and recognize gestures. My project, a storytelling 360 video is a really special case: we hope the interaction to be more intuitive and smooth, so we choose to use Kinect rather than the touch controller, but I would say that touch controller is definitely the better choice to develop interactions in VR.