LONDON Computer vision researchers at the University of California, San Diego have demonstrated techniques to improve recognition of human activity by using cameras that operate at different wavelengths than those used in human vision. The algorithms could be of use in applications ranging from surveillance, automotive safety, smart spaces and human-computer interfaces.
The multi-perspective approach involves two or more cameras observing the same person from different angles.
"The new systems we are developing are multi-perspective and multimodal," said Mohan Trivedi, professor of electrical and computer engineering in UCSD's Jacobs School of Engineering. "They allow observation of a space and occupants from various viewpoints and sense reflected as well as emitted energies. The objective is to observe and understand human movements and activities in a robust manner, and the results have been very encouraging."
Multimodal refers to more than one type of camera, for instance, thermal infrared, and color.
Details of Trivedi's work are recounted in two papers just published in the learned journal Computer Vision and Image Understanding that were co-authored with researchers in his Computer Vision and Robotics Research (CVRR) laboratory, an affiliate of Calit2 on the UCSD campus. The papers are part of a special issue of the magazine devoted to "Vision Beyond Visible Spectrum".
Trivedi says the research involving multiple modalities of sensing that the team initiated more than five years ago is yielding a lot of useful advances, including techniques that deal with new algorithms and analysis using thermal infrared video along with color video, the theme of the two papers.
The first describes ways to capture and analyze driver activities inside a car's cockpit for new types of intelligent driver assistance systems. This is supported by Volkswagen-Audi and the UC Discovery Program.
The experimental results in real-world street driving demonstrated the proposed system's effectiveness, including the tracking of the driver's head and hands regardless of the level of illumination, and fairly accurate tracking performance in noisy outdoor driving situations.
The researchers Shinko Cheng, Sangho Park and Trivedi outline a novel approach to recognizing what a motorist does while driving. "It is a major challenge to track and analyze a person's movements," explains Cheng. "This is especially true in unconstrained environments where the lighting is unreliable and where there is 'noise' in the environment because so much activity may be going on, in this case, inside and outside the vehicle."
Cheng and his fellow researchers in the smart-car lab developed a system consisting of four separate cameras and views (multi-perspective) and both thermal infrared and color cameras (multimodal). The equipment was installed on the LISA-Q, an Infiniti Q45 bedecked with cameras, sensors and processors.
The vehicle has been used in a number of automotive computer-vision experiments to date. The video-based system was then tested on the road to see how well it did with robust and real-time tracking of the driver - specifically, of the driver's important body parts (head, arms, torso, and legs).
The multi-perspective characteristics of the system provide redundant trajectories of the body parts, while the multimodal characteristics of the system provides robustness and reliability of feature detection and tracking, say the researchers. "
The second paper deals with a new approach to finding accurate correspondence between objects which are simultaneously seen by a stereo head, which uses one eye sensitive to thermal infrared, and a second to color wavelengths.
The researchers say existing algorithms do not operate well on data that has multiple objects and multiple depths that are significant relative to their distance from the camera. The new algorithm developed at UCSD offers substantial benefits, they suggest, especially in instances of close-range surveillance and pedestrian detection.
Explained Ph.D. student Steve Krotosky: "This can lead to robust and accurate pedestrian detection, tracking and analysis for active safety systems in a vehicle, and also for operating surveillance systems on a 24/7 basis."