Offers fully trained DNN for multiple sensors
MADISON, Wis. — How does a robo-car perceive the world around it — in real time — safely and accurately? If you think this is a solved problem, think again.
In an exclusive interview with EE Times, DeepScale (Mountain View, Calif.) has disclosed its unique approach to a “perception system” the startup is building for ADAS and highly automated vehicles.
DeepScale is developing the perception technology that ingests raw data, not object data, and accelerates its sensor fusion on an embedded processor.
“A good chunk of research on deep neural networks (DNN) today is based on tweaks or modifications of off-the shelf DNN,” observed Forrest Iandola, DeepScale’s CEO. In contrast, over at DeepScale, “We’re starting from scratch in developing our own DNN by using raw data — coming from not just image sensors but also radars and lidars,” he explained.
Early fusion vs. late fusion
Phil Magney, founder and principal advisor for Vision Systems Intelligence (VSI), called DeepScale’s approach “very contemporary,” representing “the latest thinking in applying AI to automated driving.”
How does the DeepScale approach — using raw data to train the neural network — differ from other sensor-fusion methodologies?
First off, “Today, most sensor fusion applications fuse the object data, not the raw data,” Magney stressed. Further, in most cases, smart sensors produce object data within the sensors, while other sensors send raw data to the main processor — where objects are produced before it is ingested into the fusion engine, he explained. Magney called such an approach “late fusion.”
Clearly, Iandola sees an inherent issue in late fusion.
It poses problems in fusing object data with raw data, he said, especially when the sensor fusion is tasked to handle multiple types of sensory data. “Think about 3D point cloud created by a lidar,” he said. “While you’re reconstructing 3D-point cloud in your sensor, you are also receiving data coming from cameras at a much different frame rate.”
While creating objects, the raw data that might have been relevant to other sensory data could be lost. Think about the moment when the sun shines directly into the vehicle camera’s lens, or when snow covers the radar, Iandola suggested. Or, when sensor data don’t agree with one another. In such a case, fusing object lists becomes challenging.
“That’s why we believe we must do raw data fusion early, not late, and do it closer to the sensors,” he said. “We think early fusion can help mitigate some of those problems.”
Next page: Designing its own DNN
Designing its own DNN
Computer vision is one area where well-established DNN frameworks already exist. Many early autonomous vehicle technology developers are leveraging them. But for other sensory data such as radar and lidar, there isn’t much trained DNN.
That’s where DeepScale hopes to come in.
It’s important to note that DeepScale has the technical chops and experience to design its own DNN.
In academia, Iandola earned his props by developing SqueezNet, a deep neural network model (DNN) model, together with researchers at UC Berkeley. Some have now joined DeepScale. SqueezNet is not designed to apply directly to automated driving problems, Iandola says, but the team developed it “with the goal of making the model as small as possible while preserving reasonable accuracy on a computer vision dataset.”
Similarly, Iandola was involved in the development of FireCaffe, a DNN framework designed to accelerate training, and enable embedded implementation. In a paper, Iandola and his team claimed that FireCaffe can successfully scale deep neural network training across a cluster of GPUs.
Asked about the DNN framework drought for radar and lidar, Iandola said, “There is a good reason for it. For a long time, the camera has been the most popular [and available] sensor that has generated a lot of data. There is enough data you can scrape from YouTube. Combined with map information, the available data makes it easier to build a DNN framework.”
Currently, DeepScale is working with a few radar and lidar suppliers to develop fully trained algorithms for OEMs. Iandola said, “Our partners include both established suppliers and new technology developers” in radar and lidar.
DeepScale’s goal is the development of DNNs that won’t necessarily require customization while ingesting data from each and every new sensor.
The startup claims that the team uses “mutual information from multiple sensors to maximize accuracy and resolve uncertainties.” Moreover, “Labeled training data can be reused across different sensor sets with minimal re-calibration.”
Magney sees a big plus in DeepScale’s claim of being “sensor agnostic,” and the DNN hosted on various processor platforms. “The DeepScale solution enables OEMs and tier ones to build out an AI-based environmental modeling solution without having to train their own networks and write their own algorithms,” he summed up.
Runs on Snapdragon?
DeepScale claims its solution is both sensor and processor agnostic. Perhaps more important, the startup stresses that it’s very processor efficient while requiring less power.
Raw sensory data coming from four cameras and one radar can be processed on a smartpone apps processor like Qualcomm’s automotive-grade Snapdragon, according to the DeepScale CEO.
Raw data from a dozen sensors could run on a single Nvidia GPU, he added.
Iandola is aware that every penny matters to car OEMs, and a low-power processor — requiring no cooling — is important to keep a vehicle reliable.
‘Rich area of innovation’
DeepScale’s idea of using raw data for sensor fusion is analogous to what Mentor Graphics has been promoting lately through its own DRS360 platform.
DRS360 consists of Xilinx Zynq UltraScale+ MPSoC FPGAs, with advanced neural networking algorithms for machine learning.
Glenn Perry, vice president and general manager at Mentor’s Embedded Systems Division, told us, “The raw data sensor fusion is not the only way to go but it is the smartest way to go” in designing a highly automated vehicle architecture.
However, the very idea of fusing raw data from multiple sensors in real time has received mixed reactions from the auto industry. One argument against it is deeply rooted in OEMs’ long history of making incremental design developments, said Perry.
Next page: Competitive landscape
Some OEMs, for example, have already invested in developing an ADAS system using object data from radar to offer adaptive cruise control, and camera/radar fusion to enable automatic emergency braking system. To suggest going back to the drawing board for a raw-data alternative is a hard sell.
On the DRS360 platform, Mentor Graphics poses a two-stage system solution — sensor fusion and perception. The platform uses a “raw-data” approach that eliminates processing from all sensor nodes. But if Mentor clients prefer pre-filtered data, Mentor can accommodate, said Perry. Mentor has developed its own algorithms and hardware-accelerated software, he added, for raw-data sensor fusion. It can provide a host of integration services built on system-support package, using Mentor IP.
Perry described that algorithms for raw-data sensor fusion — which DeepScale is obviously working on — as “a rich area of innovation.”
He believes DeepScale’s business is focused on “a key part of the solution that we’ve developed within DRS360.” To Mentor Graphics, that’s good news, said Perry, because DeepScale “could provide customers with an option for another fusion algorithm within their use of our DRS360 platform.”
So, with whom exactly is DeepScale competing?
“Well, there are a lot of things going on right now,” said Magney. “Many of the hardware companies (selling processors and/or systems) offer support for trained DNNs/CNNs. They usually offer processors/platforms plus conversion tools to support AI-based algorithms such as the ones Deepscale offers.”
Magney added, “However, most of the companies that offer hardware solutions do not offer fully trained algorithms like Deepscale.”
In Magney's opinion, “There are not that many companies that offer an AI software stack with pre-trained algorithms than can handle the total environmental model based on the fusion of RAW data.” He added, “AImotive and Mobileye have similar approaches but are specific to certain host processors.”
What about Nvidia?
Magney said, “Nvidia is major player in this space as well but they do not offer the full software stack with pre-trained algorithms for full environmental modeling.” Nvidia provides a hardware solution and tools for creating proprietary applications and also using raw-data fusion, he noted.
The analyst, however, suspects that the numbers working on an AI-based software stack are more than meets the eye. “But they are in stealth mode and not talking much about what they are doing.”
Where does DeepScale fit in the AV stack?
In a recently published presentation, VSI discussed what it calls the “AV stack” — the building blocks of autonomous vehicles. These blocks consist of five different AV domains — perception, localize & plan, decision/behavior, control and connectivity & I/O.
Asked DeepScale’s spot in the AV stack, Magney placed it in the first two blocks. DeepScale claims to support the entire environmental model including: Object Detection, Occupancy Grid, Lane Segmentation, Object tracking, Self-Localization.
DeepScale is still at a stage at which it must pick a spot in which to insert itself. Iandola doesn’t necessarily feel that DeepScale must wait for the arrival of fully autonomous vehicles to have its technology embraced. “We see the perception problem is universal. The same technology can be used for mass-production ADAS vehicles and autonomous car for mobility services.” DeepScale’s perception system, it claims, is scalable from Level 2 cars up to Level 4/Level 5 vehicles.
Deepscale also claims its stack can be fitted to existing sensor portfolios adaptable to ADAS, “allowing OEMs to add automation features during a mid-cycle refresh!” noted Magney. Deepscale believes its software could make its way into series production by 2020 or shortly thereafter.
Obviously, every autonomous-vehicle tech company claims its focus is on “safety.” But DeepScale goes beyond the platitude, declaring that “autonomous driving, by nature, is safer than human drivers.”
The topic of “safety monitor” came up when EE Times previously talked to Iandola. At that time, he suggested, “Rather than waiting for autonomous vehicles to crash, the cars themselves can self-check.” Each module would ask itself if it was a little confused when it handed off to a human driver, or ask how much confidence it had at that moment. It would send that data consistently. This sort of robotic soul-searching helps data annotators identify the hardest cases and build quality assurance models, he explained.
Magney pointed out that Deepscale “has a safety monitor built into the stack that monitors safety and captures edge cases for continuous improvement in the algorithms.” He added that Deepscale also supports over-the-air updates and has built its model around that, similar to Tesla.
— Junko Yoshida, Chief International Correspondent, EE Times