"The sense of touch is a key mechanism used by humans to learn about their world," said Okamura. "The same should be true in VR, but today designers are concentrating heavily on visual presentation. My lab will instead explore means for displaying both tactile perception through the skin and kinematic perception through the position and movement of the joints and muscles."

Perception, as opposed to sensation, holds the key to Okamura's refinement of VR accuracy, since sensations can conflict even in real life, whereas perceptions reflect a decision to "trust" an interpretation of sensations. For instance, a pencil in a glass of water appears to be broken because the visual sensation shifts abruptly where it emerges from the water.

Humans "perceive" the pencil as straight, however, because they can verify by touching the pencil that it remains unbroken where it emerges from the water, even though visual sensations suggest that it is broken. Primates, especially humans, use touch as their "perception verifier," but most lower animals follow their nose — for instance, a dog would verify that the pencil remains unbroken by smelling it.

Unfortunately, the ability to verify conflicting sensations and conjure an accurate internal perception is almost entirely missing in current VR environments. Even sophisticated electronic gloves only track the position of the hands for display in VR.

"We want to characterize materials in the real world, so that we may then mimic the necessary sensations in VR that will allow people to accurately perceive materials," said Okamura.

She will present her preliminary results and future research directions in November at the American Society of Mechanical Engineers' Dynamic Systems and Control Conference. There she will describe a method of accurately mimicking the vibrational characteristics of wood, aluminum and rubber.

"Today, even if you bump into a wall in VR, it just tells you that an obstacle exists. But in real life, when you bump into something, you use the resulting vibrational sensation to perceive the wall as made of a certain type of material," said Okamura.

Everyone admits that rapping on a wall is the best way to tell if its made of wood or just has an imitation wood grain veneer. However, no one has characterized the "output" vibrations of materials as a result of an "input" rap of the knuckles. Therefore, there are no existing models to reproduce those vibrations in VR.

"There are all kinds of physical models available to accurately represent the 3-D geometry of objects — how things look from different angles — but there are no models available to represent the vibrational signatures of materials," said Okamura.

Consequently, Okamura cut through the theoretical logjam by skipping the model for now, settling instead for recording the vibrations of real materials from sensors and then playing back the recording into a haptic display. Using real human subjects to experience the recorded vibration sensations, Okamura was able to accurately induce the perceptions of wood, aluminum and rubber.

"We allowed the test subjects to select the best parameters for our vibrational recordings by comparing them with vibrations from real materials," said Okamura.

Subjects used a tethered stylus (from San Jose-based Immersion Corp.) to "tap" on real wood, aluminum and rubber blocks for direct sequential comparisons with nine different parameter sets of the real recordings. The parameters of the recordings were variables such as a multiplier for amplitude of the vibration waveform corresponding to the impact velocity of the stylus. Other parameters besides the amplitude-velocity slope included the decay rate and frequency parameters of the recordings. All test subjects were able to exceed 80 percent accuracy in identifying virtual materials of wood, aluminum and rubber after about 15 minutes of parameter selection.

Tele-operated medical surgery is one area where the new virtual reality system could find an immediate application. "There are many tactile problems to be solved for tele-operated surgery. For instance, when you put a needle through the skin it is hard to avoid overshoot; you have to compress the skin to get the needle through, but once it's through you have to back off the pressure in just the right manner to end up with the needle at the right depth," said Okamura.

Solving the needle insertion problem, however, would involve characterizing the soft tissues of the human body — a difficult task that illustrates how far haptics research needs to travel from characterizing the simple vibrational characteristics of wood, aluminum and rubber. Consequently, for the foreseeable future, Okamura's lab will perform basic research into what she calls "dexterous manipulation."

Ultimately, multiple robotic fingers will cooperate to grasp, manipulate and explore the world, for the same reason that humans verify conflicting visual sensations.

"You have to work backward from the object to the actuators that will manipulate it when formulating a dextrous manipulation problem for a robot," said Okamura. In doing so, a system of equations can be built that describes the robot-object system in terms of shared characteristics, such as points of contact, finger-arm locations compatible with object geometry, and kinematic bounds such as how much pressure can be applied when picking up an egg.

The first step in dexterous manipulation is usually to calculate backward from the desired pressure on the object, to the fingertip forces to be employed by the robot. From there things get complicated, since accuracy in anthropomorphic robotic hands involves rolling and sliding them against the object as it is manipulated. Calculations take as input the location, texture and relative velocities of object and fingertip, and output parameterized velocities to describe the motions of the contact frames over the fingertip and object surfaces.

While other labs have explored most of these dexterous manipulation problems, Okamura believes her effort will have unique aspects that pull together diverse multidisciplinary resources to solve the problem. Automated grasping and motion planning, for 3-D manipulation of realistic virtual materials with rolling and sliding, will be the ultimate goal of the lab. Today such calculations must be done offline, but Okamura believes that better algorithms can soon solve the problem.