Design Article
Processing algorithms allow seamless low-distortion automotive 360° video imaging
Akio Nezu, Fujitsu Semiconductor America
12/16/2010 9:32 PM EST
Studies have found that cameras are the best way to improve visibility around a vehicle. There has been growing global interest in car-mounted cameras to improve the driver’s awareness of surrounding and make driving safer. Not surprisingly, vehicle-mounted cameras have grown in popularity as a tool for enhancing driver safety—and are also facing proposed legislative mandates in many countries.
Drivers navigate their vehicles in a variety of situations, such as parking, turning, and merging, that demand immediate visual checks of the vehicle's perimeter. But the current reversing and peripheral vision cameras demand too much of drivers. With conventional technologies, each camera shows a different perspective and sightline. Because the display changes instantly, the driver must be able to instantly recognize which view is being presented, which can be difficult.
The underlying technologies behind such systems are conventional, multi-camera, “bird’s-eye-view” technologies (see figure below) that stitch together two-dimensional images showing a vehicle’s perimeter from front, side, and rear cameras. The resulting “top-view” images typically are distorted or have poor stitching at the seams. These images do not provide enough detail about the surrounding area, and drivers could easily miss something important—objects and pedestrians.
These bird’s-eye-view systems' conventional technology-based images do not meet the need of reducing the burden on the driver for visual checks in the driver’s field of view.
In addition to the driver’s own direct-eye field of view, rearview, and sideview mirrors, as well as rearview monitors, enhance the field of view. Although these features can be used to quickly cover a great field of view, the need to instantly refer to all of them imposes a great cognitive load on the driver.
The four-camera bird’s-eye-view systems (figure below) only provide a video of the roadway within about two meters of the vehicle, necessitating a rearview monitor. These cameras do not integrate the field-of-view information the driver requires, thus failing to adequately reduce the cognitive load on the driver.
The conventional technology also falls short of helping the driver recognize the point of view (perspective), sightline, and field of view shown on the monitor.
With conventional technologies, each camera and each function differs in perspective and sightline. Because the display changes instantly, the driver must be able to instantly recognize which view is being presented. This makes it more difficult to make a perimeter check and limits the situations in which the technology can be used effectively. In addition, it takes a long time to get used to such systems.
Drivers navigate their vehicles in a variety of situations, such as parking, turning, and merging, that demand immediate visual checks of the vehicle's perimeter. But the current reversing and peripheral vision cameras demand too much of drivers. With conventional technologies, each camera shows a different perspective and sightline. Because the display changes instantly, the driver must be able to instantly recognize which view is being presented, which can be difficult.
The underlying technologies behind such systems are conventional, multi-camera, “bird’s-eye-view” technologies (see figure below) that stitch together two-dimensional images showing a vehicle’s perimeter from front, side, and rear cameras. The resulting “top-view” images typically are distorted or have poor stitching at the seams. These images do not provide enough detail about the surrounding area, and drivers could easily miss something important—objects and pedestrians.
These bird’s-eye-view systems' conventional technology-based images do not meet the need of reducing the burden on the driver for visual checks in the driver’s field of view.
In addition to the driver’s own direct-eye field of view, rearview, and sideview mirrors, as well as rearview monitors, enhance the field of view. Although these features can be used to quickly cover a great field of view, the need to instantly refer to all of them imposes a great cognitive load on the driver.
The four-camera bird’s-eye-view systems (figure below) only provide a video of the roadway within about two meters of the vehicle, necessitating a rearview monitor. These cameras do not integrate the field-of-view information the driver requires, thus failing to adequately reduce the cognitive load on the driver.
The conventional technology also falls short of helping the driver recognize the point of view (perspective), sightline, and field of view shown on the monitor.
With conventional technologies, each camera and each function differs in perspective and sightline. Because the display changes instantly, the driver must be able to instantly recognize which view is being presented. This makes it more difficult to make a perimeter check and limits the situations in which the technology can be used effectively. In addition, it takes a long time to get used to such systems.
|
|
|
|
|
Navigate to related information
Rick DeMeis
12/16/2010 10:36 PM EST
This is interesting technology, but it is up to the Tier 1s and OEMs to implement it in a useful display that doesn't distract the driver.
Sign in to Reply
thinkski
12/19/2010 4:37 PM EST
Security and surveillance applications were briefly mentioned, but the market may be larger than implied. A 360-degree view of a vehicle leading up to an accident could be useful to insurance companies. Or perhaps someday Google StreetView will include live streaming video, captured in real-time from a mesh/5G network of vehicles equipped with 360-degree view systems. That would be Orwellian, but cool! People tend to behave better when they know they're being watched, in general.
Sign in to Reply
Rich Krajewski
12/20/2010 9:27 PM EST
As a consumer, this is something I'd pay extra for, mostly for the security and insurance value, but also for the real-time information about blind spots. Combined with infra-red, it would be a great tool for night-time driving.
The hardware implementation seems easy enough for a tech to do. It's the image integration from several cameras that would be the only relative hurdle. Am I wrong in thinking the hurdle is not that big? (I'm not an expert in image processing.)
Sign in to Reply
K1200LT Rider
12/21/2010 9:50 AM EST
This system would be great if it could switch between or combine visible and infra-red images. I'd like it on my vehicles, but I'm trying to figure out where all the cameras would be mounted to get a clear view without obstructions. Most rear windows have those horizontal heating elements in them. So, are the cameras mounted externally? If so, they'll have to be cleaned regularly if they don't have some kind of a fancy little self-cleaning mechanism. Lots to think about and overcome in real-world use. About the image integration hurdle: I think blending images well and then creating a virtual 3-D view would be quite difficult to get working correctly if you are starting from scratch on your own. I think you would have to be quite good at image processing and have a knack for figuring out what clues in the images to use for the 2D-to-3D algorithms. Since each camera installation is unique (relative camera positioning and aiming), each system would have to go through a calibration of some sort, so coming up with the calibration process alone might be very challenging.
Sign in to Reply
Luis Sanchez
12/22/2010 4:55 PM EST
Wow! I'm impressed! This really looks useful and perhaps this might become a good seller soon.
Good question from Rich K. Is the image processing difficult? I suppose the algorithms aren't that complex but the processing hardware must be powerful in order to complete a merge of the images in real time for the viewer.
But perhaps 4 cameras wouldn't be the best option and cost wise 3 might be enough... consider the blind spots only.
I'll be on the watch out for this and remember I saw it on EETimes first.
Sign in to Reply
George.Bitar
1/26/2011 10:37 AM EST
Common usage of this technology is inevitable, for driving. Commercial vehicles already run DVR for liability. And this algorithmic offering pushes past the bird's eye by a quantum leap, even factoring in driver distraction. Consider that "drive by wire" is firmly entrenched (potentiometer as accelerator control), and relatively speaking, I would adapt to camera use, psychologically long before I'd get rid of physical brake linkage ("brake by wire"). People are going to start driving only by viewing the camera, as they text. That is, until sufficient networking between vehicle and comm devices prevents this kind of thing. It's all coming.
Sign in to Reply