Kinect Air Guitar Prototype
In 2007 I attempted to make a simple Air Guitar prototype using a stereo camera (watch video). Unfortunately it didn’t work very well. You had to stand very still and the depth image wasn’t as clean as I would like.
Now that the much anticipated Xbox Kinect camera was out and opening up by the way, I wanted to port this demo to explore what the potential of interactive music games could be in the future & also just to experiment with the Kinect. So here is the outcome so far…
Technical Details
Written in c++ using openFrameworks and openCV for image processing. Using the ofxKinect addon and the libfreenect. Thank you to the openframeworks and openkinect communities for enabling this to happen. A big thank you to Microsoft for bringing this technology to the mass market.
How it works
First it thresholds the scene to find a person, then uses a histogram to get the most likely depth of a person in the scene. Then any pixels closer than the person to the camera are possible hands. It also uses contour extremity finding on the person blob to look for hands in situations where your hand is at the same depth as your body. It only works if you are facing the camera front on. Then it uses one hand as the neck of the guitar, drawing a virtual line from the neck through the person centroid to create the guitar line. The other hand is tracked to see if it passes through this line, strumming the guitar. The neck hand position controls the chord.
Limitations
My hand tracking is a simple first attempt, its quite inaccurate and a bit flaky. You also have to face the camera in quite a specific way. With Xbox Kinect games, they have full skeletal tracking. Now with OpenNI, us homebrew developers can also play with these ideas. With full 3d skeletal tracking this would be 100% better. Also in terms of sound it is very simple chords, and of course is no replacement for a real guitar, but this is more like Guitar Hero / Rock Band playing than accuracy.
The Future
To clarify, I am making this to experiment with the Kinect and full body interaction. I’ve no plans to attempt to bring this to market in any way. I am sure there are companies out there doing this already.
Related Works
Since creating the demo I’ve seen this virtual air guitar project (2004/2005) by a team in Helsinki. They use a normal webcam to track orange gloves for playing the guitar, with a more advanced sound engine. They have recently launched formally as the Virtual Air Guitar Company.
Earlier this year someone created Theremin Hero Air Guitar, as the name suggests, you can play Guitar Hero using the theremin, which sort of looks like air guitar by the way you hold your hands.
