MIT gives LCDs a hand in recognizing gestures

PORTLAND, Ore. — Gesture recognition today requires a 3-D image sensor that can detect depth. But researchers at the Massachusetts Institute of Technology claim that the addition of a layer of photodiodes to the backside of a liquid-crystal display lets the LCD recognize hand gestures made in front of the screen.

The technique promises in-air gesture control of everything from mobile phones to flat-panel TVs, according to the MIT researchers, who will demonstrate their work in Yokohama, Japan, this weekend at Siggraph Asia (Dec. 16-19).

The system is called a bidirectional screen, because light goes into it as well as comes out of it. Most of the time the LCD operates as usual, coloring light that passes through it from the backlight. But periodically, and too quickly for humans to perceive, the LCD switches to become an array of tiny apertures, allowing light to enter and form images on the photodiodes from slightly different angles. That information can then be used to calculate the 3-D depth and enable gesture recognition.

"We create what's called a light field array," said MIT research scientist Henry Holtzman. "The original experiment used pinholes, but that let less than 1 percent of the light through. What we use now is a spatial heterodyne pattern that lets through 50 percent of the light—half white pixels and half black pixels—but in a pattern that makes it easy to reconstruct the original image."

Holtzman performed the work with MIT professor Ramesh Raskar, visiting researcher Douglas Lanman and doctoral candidate Matthew Hirsch.

MIT replicated a 361-pixel (19 x 19) pattern across the screen for the demonstration at Siggraph, but different sizes and configurations of the spatial heterodyne pattern would be crafted for different-sized screens and different applications, the researchers said.

Earlier this year, Sharp announced an optical sensor LCD pad that displays the names of buttons and other legends on the touchpad, then senses a finger's presence with photodiodes that are integrated next to the thin-film transistors for each pixel. MIT's innovation, for which it is seeking a patent, separates the TFTs' emitter layer from the photodiodes' detector layer. Changing the spacing between the layers enables control of how far in front of the screen gestures can be made and recognized.

Sharp's optical LCD only senses fingers on its surface, because there is no separation between the TFTs and the photodiodes. For the Siggraph demo, MIT adjusted its simulated 20-inch display to recognize gestures from up to 20 inches away, though for a mobile phone application that gap would be narrowed to sense gestures made just an inch or two in front of the screen.