3D Modeling Gets a Boost from Kinect

Not all hackers are trying to steal credit-card information or personal data. Many of them, in fact, are helping society. They are developing open-source code for video-game interfaces such as Microsoft’s Kinect. The resulting applications are often innovative ideas the folks in Redmond, Wash., never imagined.

Kinect is designed to replace the game controller that normally plugs into the Xbox 360. It lets users play computer games such as volleyball using onscreen avatars they control by just moving their own bodies. The device’s cameras, sensors, and software let it detect movement, depth, and shape and position of the human body.

But Kinect hackers have taken the device far beyond its gaming roots. Programmers such as computer scientist Oliver Kreylos from the University of Calif., Davis, have no interest in playing gesture-based video games. Instead of plugging a Kinect into an Xbox, he hooks it to a computer. The Kinect’s “eyes” are actually a pair of cameras. One camera detects depth and another picks up colors. Each uses a metal-oxide semiconductor image sensor. Kreylos converts Kinect into a 3D camera by combining depth and color-image streams and projecting them into 3D space. Data reconstructed in such a way look like real 3D objects inside the camera field-of-view. The result is what Kreylos calls a 3D holographic image.

Kreylos says he had been looking for inexpensive cameras like those in Kinect for visualizing scientific data. He contacted PrimeSense in Israel which developed the original technology that later became Kinect. “Unfortunately, this happened shortly after PrimeSense had entered an exclusive agreement with Microsoft,” he says. “So although I could not purchase the cameras back then, I have known exactly what Kinect could do for the last two years or so.”

Kinect’s depth camera is paired with a near-infrared laser projector, which spreads a pattern of dots over the room and its occupants. The camera detects this pattern and, based on differences between preset and observed pattern, calculates the distance to each visible dot. From this data, it builds a “depth image,” which contains a distance value for each pixel. This method is a variation of the well-known “structured light” approach in which a scene’s 3D structure is uncovered based on distortions in a projected pattern. Color is added to the 3D model via RGB input from the image sensor.