Kinect AR with FFD

20 April, 2015


The application developed combines both aspects of virtual and augmented reality using the Kinect and a computer as the platform of choice. The application uses data from the Kinect to map the real world into a virtual and three dimensional environment. Moving in the real world correspondingly changes the position and depth of the player in the virtual environment. A 3D geometric model, bounded by control points, is then projected into the virtual world. Since the model in virtual space has similar positional coordinates to the real world, it gives the player an impression that the model lies within the real world. The model can then be interacted with using the player’s hands as the interface. Moving the hands, the player can select and move the control points (which lie in the virtual space) which in turn deforms the model.

Cross Platform Framework

The application was written using a cross platform framework written in C++ 1114. This framework was built from scratch and is currently under development.

The original intention of this framework is to be adapted into a cross platform engine. However I’ve started small, so at first this is just a cross platform graphics framework. This is a cohesion of different components I’ve developed in the past as well as a lot of new stuff.

This framework is being created to keep one useless objective in mind: On Windows, there should be an option to switch from DirectX to OpenGL at runtime without any obstruction/interference or application restart. Yes this feature is pointless but I’m doing it just because I can.

With the release of Visual Studio 2015 which adds support for more features of C++ 1114, this framework is now being reworked to also add support for an offline shader pipeline.

Kinect AR Rendering

Each “pixel” in the scene is a voxel, i.e. a cube which is being rendered using mesh instancing. The depth of the pixel is calculated in the vertex shader using the depth texture output by the kinect.

The colour of each voxel is added in the pixel shader, which uses the colour texture data from the Kinect.

The model is loaded at runtime using the FBX SDK.

The FFD technique in use is the Sedberg-Parry technique, calculated using the trivariate bernstein polynomial.