Breaking News
News & Analysis

Toyota Case: Single Bit Flip That Killed

10/25/2013 03:35 PM EDT
104 comments
< Previous Page 2 / 3 Next >
More Related Links
View Comments: Newest First | Oldest First | Threaded View
<<   <   Page 9 / 11   >   >>
DrQuine
User Rank
CEO
Noise and probabilities
DrQuine   10/26/2013 7:46:01 PM
NO RATINGS
With the numbers of miles cars are driven and the large number of engine cylinder operations per mile (say 12,000 for a 6 cylinder car being driven at 60 mph at 2,000 rpm), low probability problems are likely to surface. That said, in electrically noisy environments with connectors subjected to adverse conditions and wires flexing, the possibility that a bit might get flipped doesn't seem surprising. I guess the question becomes: what processes enable engines to quickly return to proper operating modes when errors are detected? I know for sure that problem hasn't yet been solved on my home computer.

Bert22306
User Rank
CEO
Re: Securing future for automobile electronics control
Bert22306   10/26/2013 6:15:40 PM
NO RATINGS
"There is in my opinion no way of 100% preventing an uncommanded wide open throttle condition occurring from time to time ..."

Yes, this is all about probabilities, and btw, exactly the same holds for mechanical throttles. I myself had this occur to me, in the pre-electronic car control era (well, not full throttle, but certainly open throttle).

It should not be too difficult to design throttle controls that only give the command for a short period of time, which is the way we tend to do this sort of control. E.g., you read the throttle or other command signal at, say, 10 Hz. If the signal is not consistent for n hits, you either go to an alternate source, or you fail safe. If updated commands are not received by the output process, again you fail safe. If the output process dies, again the throttle controller fails safe.

I've had one odd situation in which the valves we were controlling would slowly cycle open with only a single discrete command. So EVEN IF that single discrete command corrected itself after 100 msec, the valve would continue to cycle to full open, before laboriously beginning the close cycle. A very unfortunate combination of events. Since changing the valves appeared to be impossibly difficult, we ended up dramatically improving the error detection logic before closing any discrete signal to valves, which solved the problem (well, at least for several 10s of thousands of years, doing the statistical analysis).

I suspect that the Toyota throttle issue might be caused by a similar combination of unfortunate coincidences.

JCreasey
User Rank
Freelancer
Toyota's culpability here is the tip of the iceberg for everyone
JCreasey   10/26/2013 4:36:42 PM
One assumes that the hardware and software gaps identified in this ECU implementation is sobering for anyone implementing driver assistance systems, drive by wire controls as well as those pushing fully autonomous cars.

Most automobiles today have multiple cpu's communicating over non-redundant networks with a non-zero error rates; as the complexities of control increase in drivers assisted, drive by wire and autonomous vehicles the ability to fully test and verify the implementation reduces. Even though the software and hardware may be designed to meet strict standards, the standard themselves have limitations and are open to some interpretation.

My major concern is that automation systems built  by multiple organizations have an almost zero chance of reaching the same decision in the same time in any given emergency  (and faults are just another emergency) event. Putting lots of these systems in close proximity in high speed flowing traffic may be a classical Butterfly Effect when things do fowl up.

We have so many things to fix to make these systems safe, and it is not helped by the performance achieved by some vehicles. Who on earth needs 0-60 in under 4 seconds  where unintended acceleration means the vehicle may be pulling 0.7g from a standing start. With human perception/reaction times in the 0.5 seconds range at best (for a fully attentive driver to brake as here: http://www.ecu.edu/cs-dhs/ot/upload/AOTA_Brake_Reaction_Poster.pdf ) the vehicle moved about 11ft before you had any chance to react. (Just as disconcerting and deadly would be un-intended braking at highway speeds on those around your vehicle).

Big red buttons for emergency disconnects, brakes expected to dissipate the full engine power or any other human activated device you care to describe will not help. The reaction time of the automation will always be far shorter than any human reaction, so putting reliance on the human to be the arbiter in case of emergencies IMO verges on asinine.

jimlux
User Rank
Rookie
is it worse than mechanical springs
jimlux   10/26/2013 4:19:07 PM
For all this talk of unuusal software failures, one should recall that broken throttle return springs on a mechanical linkage are by no means unheard of. I've had one, several of my friends have, back in the 70s and 80s.

All this talk of "sudden acceleration" and the ensuing potential carnage is really a comment on appropriate driver training.  A 70s muscle car going wide open throttle is going to accelerate pretty quickly, and you don't have the saving grace of a modern rev limiter, so the "shift to neutral" aspect is likely to wind up wiht engine damage.

Back then, the argument for neutral was "you'll lose your power steerng and power brakes" both of which are bogus.  You're not going to be parallel parking, you're steering to the side of the road, and the turning effort isn't all that high when the power steering pump is off: People do have the belt fail or the pump fail, and nobody leaps up to sue because the car was uncontrollable.  Likewise, power brakes are an assist, and vacuum operated.  With the engine at WOT, there's not much vacuum anyway.  In any case, you can still stomp on the brake and bring the car to a stop.

The real reason for the shift to tneutral advice is because people would turn the switch to the LOCK position and remove the key, and then be unable to turn the steering wheel. 

In any case, this is just a matter of appropriate training.  Everyone should have the instructor turn the engine off while driving, so they know what it's like.  My driving instructor did this, my father did it. 

 

Now lets talk about "I stood on the brake and it didn't stop".  There are lots of documented cases where drivers have believed they were on the brake, and were on the gas instead. But leaving that aside, there is no car made where the brakes cannot overpower the engine, IF and only IF, you apply the brakes hard.   Again this comes back to trianing. if you try to "ride the brake" to slow down (as opposed to stop), yes, you will overheat the brakes and they lose their effectiveness.

 

 

 

Gordon.Apple
User Rank
Rookie
Software Errors Can be Subtle
Gordon.Apple   10/26/2013 3:05:09 PM
NO RATINGS
I have analyzed systems many times, where a minor, unnoticed software error could prove catastrophic.  In one particular case, the probability of an incremental error happening was extremely small.  However, the cumulative effect over a long period of time eventually proved catastrophic to the system's operation. The company had spend hundreds of thousands of dollars testing, in an attempt to figure out what was happening.  I was brought in as an independent cousultant for another purpose, but needed to examine the relevant code to do my performance analysis. In my final report, I mention the errors I had found in the code, and their potential impact.  The rresponse was, "OMG, we've been trying for months to figure out what was going wrong."  I billed them heavily that month and they gratefully paid it.

Toyota could have used my expertise.

Antony Anderson
User Rank
Rookie
Re: Securing future for automobile electronics control
Antony Anderson   10/26/2013 2:43:17 PM
NO RATINGS
There is in my opinion no way of 100% preventing an uncommanded wide open throttle condition occurring from time to time somewhere in a world population of over 1000 million vehicles. However the effects of a wide open throttle could be largely prevented by means of a totally independent fail-safe,  a kill switch for example, that reduces engine power in an emergency. The present situation in which drivers have to brake against full engine power is totally unnecessary and potentially very dangerous. From a functional safety point of view it is unacceptable to make the driver the fail-safe for the malfunctioning electronics.

 Hot News for 1897 – Preventing runaway electric taxis

 Improved electric hansom cabs introduced by the Electric Vehicle Company in New York powered by  two Westinghouse motors of 1.5 kW and 8oo RPM.

An emergency button operated by the driver's heel, could render the whole system powerless

Moms,Gijs: The Electric Vehicle –Technology and Expectations in the Automobile Age

John Hopkins University Press 2004 page 83

 So much for progress!

wilber_xbox
User Rank
Manager
Re: Securing future for automobile electronics control
wilber_xbox   10/26/2013 12:05:11 PM
NO RATINGS
it shows how dangerous an engineering or software engineer can be. Do not frustrate them and keep them happy (pun intented).

_hm
User Rank
CEO
Securing future for automobile electronics control
_hm   10/26/2013 9:13:45 AM
NO RATINGS
This may be true for other auto maker. The speed at which new technology come to market, they are also prone to similar error. What are the steps suggested to prevent future errors like this. Is it really possible to prevent it 100%?

Antony Anderson
User Rank
Rookie
Re: Single bit flip
Antony Anderson   10/26/2013 4:28:41 AM
NO RATINGS
Yes ISO26262 is correct standard and it does derive from IEC 61508. you may find the following open letter to the NAS Committee on Sudden Acceleration written in Nov 2010 a useful starting point.

"The poor state of functional safety in the Automobile Industry" by Armstrong, Kirk and Anderson www.insidefunctionalsafety.com/article/31.html‎ 

 

 

 

Sanjib.A
User Rank
CEO
Re: Single bit flip
Sanjib.A   10/26/2013 2:47:53 AM
NO RATINGS
One of the safety standards for automotive electronics systems is ISO 26262, which was adapted from the standard IEC 61508, a popular safety standard followed in the industry for "Functional Safety" of programmable electrical and electronics system. I have worked on a number of safety programs per IEC 61508. From my experience, the possible causes of failures: such as  "unprotected critical variables", "...tasks can die without the watchdog resetting the processor", erroneous bit-flip undetected etc. could have been averted if the embedded design of the throttle control system was compiled as per the safety standard. Unfortunately ISO 26262 was introduced after 2007 (most probably in 2010-2011 time frame) and I guess it was not mandatory for the automotive industry to comply with IEC 61508 for the automotive electronics system safety before that.

 

<<   <   Page 9 / 11   >   >>
Flash Poll
Radio
LATEST ARCHIVED BROADCAST
EE Times editor Junko Yoshida grills two executives --Rick Walker, senior product marketing manager for IoT and home automation for CSR, and Jim Reich, CTO and co-founder at Palatehome.
Like Us on Facebook

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)
EE Times on Twitter
EE Times Twitter Feed
Top Comments of the Week