Basement Projects is a new section on my blog that I’ll use to share some of the projects in create during spare-time in my virtual basement.
This week’s project is all about Head Tracking and how it can be used to navigate in a 3D environment.
Download Video
Last week, I stumbled across an article that uses Head Tracking to interact with video games. It immediately grabbed my attention so I began to search around on how I could implement this in my own programs. The Head Tracking API used in Torben’s project is provided by Seeing Machines. They have a Non-Commercial License available that provides exactly what I wanted to take advantage off.
Before I get started I want to point out that my knowledge of 3D math is very basic. So if you see any formulas that don’t quite add up, drop me a message if you will and I will correct it. Also, the 3D engine I use in this project is probably not what you want for writing a full 3D video game, but since I have no experience with XNA or anything of that sort, I’ll stick with 3D WPF.
The ManagedFaceAPI Project
To keep things simple, I’ve written the Managed FaceAPI Wrapper library using Managed C++ code. I could have easily written a C# interop library that exposes the same functionality but I find that interacting with legacy code is a lot more straight forward with Managed C++. The library exposes the Engine class which is a wrapper for the FaceAPI C-calls.
The code used in Start method is just a copy-paste of the code found on the bottom of the FaceAPI Specifications page. Same goes for the Stop method.
void ManagedFaceAPI::Engine::Start(bool showVideoDisplay)
{
// Ensure that only one engine is active
if(Current != nullptr)
return;
// Save a reference for the current engine
Current = this;
// Register a callback function to receive log messages
smLoggingRegisterCallback(0, receiveLogMessage);
smAPIInit();
// Register windows driver model (WDM) cameras
smCameraRegisterType(SM_API_CAMERA_TYPE_WDM);
// Create a new Head-Tracker engine
smEngineCreate(SM_API_ENGINE_LATEST_HEAD_TRACKER,&engine);
// Register a callback function to receive the tracking data
smHTRegisterHeadPoseCallback(engine, 0, receiveHeadPose);
// Fine-tune the engine
smHTSetTrackingRanges(engine, 0.10f, 0.8f);
smHTSetRestartThreshold(engine, 30.0f);
// Create and show a video-display window
destroyVideoWindow = showVideoDisplay;
if(showVideoDisplay == TRUE)
smVideoDisplayCreate(engine,&video_display,0,TRUE);
// Start tracking
smEngineStart(engine);
}
void ManagedFaceAPI::Engine::Stop()
{
// Destroy engine
smEngineDestroy(&engine);
// Destroy video display
if(destroyVideoWindow == TRUE)
smVideoDisplayDestroy(&video_display);
smAPIQuit();
// Reset the current engine reference
Current = nullptr;
}
Notice that I keep a static reference to the Engine class, ensuring that the unmanaged loopback function that receives the Head Tracking coordinates can pass them to the current Engine.
Next we’ll take a look at the receiveHeadPose function. This is the heart of our library. This loopback function will be called by the FaceAPI when the tracking engine is running, passing along the position of the head in 3D space, along with its 3D rotation and a confidence factor that indicates how much the values can be trusted.
void STDCALL receiveHeadPose(void *,smEngineHeadPoseData head_pose, smCameraVideoFrame video_frame)
{
ManagedFaceAPI::P3D ^headPosition = gcnew ManagedFaceAPI::P3D();
ManagedFaceAPI::P3D ^headRotation = gcnew ManagedFaceAPI::P3D();
headPosition->X = head_pose.head_pos.x;
headPosition->Y = head_pose.head_pos.y;
headPosition->Z = head_pose.head_pos.z;
headRotation->X = head_pose.head_rot.x_rads;
headRotation->Y = head_pose.head_rot.y_rads;
headRotation->Z = head_pose.head_rot.z_rads;
ManagedFaceAPI::Engine::Current->FireHeadPoseChanged(headPosition, headRotation, head_pose.confidence);
}
As you can see, all this function does is wrap the coordinates it receives and passes to the current Engine class’s FireHeadPoseChanged method. This method will invoke the HeadPoseChanged event of type HeadPoseChangedDelegate.
void ManagedFaceAPI::Engine::FireHeadPoseChanged(P3D ^headPosition, P3D ^headRotation, float confidence)
{
HeadPoseChanged(headPosition, headRotation, confidence);
}
That’s it for the unmanaged part of the project. On to the managed code :-)
The ManagedFaceAPIDemo Project
This is a WPF (Windows Presentation Foundation) project with a single window, MainWindow. Before jumping in the code, let me show you how I got the 3D world you see in the demo.
Sacred place 3D Scene
Since my 3D modeling skills are basically zero, I began to search around for readily available XAML 3D scenes. As expected, my search came up empty. So I needed to find a 3D program that is able to export some of the more popular file formats to XAML. A post on the XNA forums suggested a program called DeleD Lite for creating 3D game levels. After installing DeleD Lite I noticed it containing a sample called SacredPlace. This 3D scene was exactly the kind I wanted in my demo:
Unfortunately, DeleD LITE has no export functionality to XAML. I’ve hit another wall. A bit of searching brought me to a program called 3D PaintBrush. One of its features was exactly what I was looking for:
Create content for Microsoft Expressions Blend
Export your 3D models (with animations) to XAML for using in Microsoft Expressions Blend
Wow, that’s a huge step forward. Now to bridge the gap between DeleD Lite and 3D PaintBrush. One of the common file formats between the two is Wavefront OBJ:
One Export/Import later I am where I was hoping to be:
Just one export away from XAML as shown in the following screenshot:
Now that I have my 3D scene, I can start integrating it in my WPF application.
The Demo Application
In the XAML code of the MainWindow, I added the Viewport3D control. I then split up the 3D Scene so that most of the 3D mesh data is in another file in my project since its rather large. This leaves my 3D Viewport with the following code:
<Viewport3D>
<Viewport3D.Resources>
<ResourceDictionary Source="Scene.xaml" />
</Viewport3D.Resources>
<!-- ModelVisual3D Data goes here -->
</Viewport3D>
After watching the scene render correctly, it was time to start manipulating the camera according to my head position. I therefore defined two dependency properties, one for the camera position and one for the direction I am looking at.
public static readonly DependencyProperty PositionProperty = DependencyProperty.Register("Position", typeof(Point3D), typeof(MainWindow), new UIPropertyMetadata(null));
public static readonly DependencyProperty LookDirectionProperty = DependencyProperty.Register("LookDirection", typeof(Vector3D), typeof(MainWindow), new UIPropertyMetadata(null));
These properties can now be bound to the camera on the Viewport3D control.
<Viewport3D.Camera>
<PerspectiveCamera Position="{Binding Position}" LookDirection="{Binding LookDirection}" FieldOfView="45.0" />
</Viewport3D.Camera>
The constructor of the MainWindow starts the FaceAPI engine:
engine = new Engine();
engine.Start(true);
engine.HeadPoseChanged += new HeadPoseChangedDelegate(Engine_HeadPoseChanged);
The argument passed to the Start method indicates that we also want to view the FaceAPI webcam video data.
The Engine_HeadPoseChanged event handler is then called every time the FaceAPI detects a new head position in 3D space.
private void Engine_HeadPoseChanged(P3D headPosition, P3D headRotation, float confidence)
{
Dispatcher.BeginInvoke(new Action(() => { Confidence = Convert.ToInt32(100.0 * confidence); }));
if (confidence == 0.0)
{
if (!isSearching && !whereAreYou)
{
whereAreYou = true;
Dispatcher.Invoke(new Action(() => { whereRU.Visibility = Visibility.Visible; }));
}
return;
}
if (isSearching)
{
isSearching = false;
Dispatcher.Invoke(new Action(() => { searching.Visibility = Visibility.Collapsed; }));
}
if (whereAreYou)
{
whereAreYou = false;
Dispatcher.Invoke(new Action(() => { whereRU.Visibility = Visibility.Collapsed; }));
}
Dispatcher.BeginInvoke(new Action(() =>
{
var rX = RadianToDegree(headRotation.X);
var rY = RadianToDegree(headRotation.Y);
rX *= -1; // Invert axis
rY *= 2.5; // Faster horizontal movement
if (compensateRotation)
yRotateCompensation += rY / 2;
rY += yRotateCompensation;
Transform3DGroup transformGroup = new Transform3DGroup();
transformGroup.Children.Add(new RotateTransform3D(new AxisAngleRotation3D(new Vector3D(1, 0, 0), rX)));
transformGroup.Children.Add(new RotateTransform3D(new AxisAngleRotation3D(new Vector3D(0, 1, 0), rY)));
var now = DateTime.Now;
BeginAnimation(LookDirectionProperty, new Vector3DAnimation(transformGroup.Transform(new Vector3D(Position.X, Position.Y, Position.Z)), TimeSpan.FromMilliseconds((now - lastFrame).TotalMilliseconds * 5)));
lastFrame = now;
}));
}
This is the heart of our demo application. The flags at the beginning of the method (isSearching, whereAreYou) provide feedback to the user. The confidence property set on the first line is shown in the top-right corner of the MainWindow at runtime. After performing some transformations to the X and Y axis we finally animate the LookDirection property to the new direction. The animation is there to smoothen the movement from one position to another.
Now that we are able to look around, let’s focus on movement. I wanted to move in the direction we are looking, so I had to brush up on my 3D math. At first I wrote the code in the HeadPoseChanged method, but that slowed things down a lot. So I decided to split up the code in another thread that will check and update the position at regular intervals:
private void Timer_Tick(object sender, EventArgs e)
{
var lookDirection = LookDirection;
if (moveForward && lookDirection != null)
{
lookDirection.Normalize();
lookDirection = Vector3D.Multiply(lookDirection, 5);
var newPosition = Vector3D.Add(new Vector3D(Position.X, Position.Y, Position.Z), lookDirection);
Position = new Point3D(newPosition.X, /* Discarding Y-value, unless you want to fly */ Position.Y, newPosition.Z);
}
}
That’s all there is to it !
Getting The Code
As usual, I’ve uploaded the code to my SkyDrive account. Note that in order to compile it, you will have to download the FaceAPI from the SeeingMachines website. There is also a binary version available for download, which should run just fine when you have a webcam installed. Make sure to install the prerequisites as well, as they contain the redistributable binaries for the FaceAPI.
Source
Prerequisites and Binaries
Final Thoughts
What began as a random idea turned out to be one of the most amusing basement projects I’ve created so far. The fact that I was able to accomplish what I had in mind with so little code was very rewarding. The whole project took me just over a day from idea to a working proof of concept. Also, if I had known that an API as FaceAPI existed for quite some time now, I would’ve created this much sooner. As a final note, I would just like to point out that at some times I became a bit dizzy from the effect on screen, so if you decide to play around with your own Head Tracking project, be prepared to take a break :-)
