REGISTER | LOGIN
Breaking News
Blog

Why 'Toyota's Killer Firmware' May Not Have Been a Killer After All

NO RATINGS
View Comments: Threaded | Newest First | Oldest First
Kevin Neilson
User Rank
Author
Brakes
Kevin Neilson   3/9/2017 12:35:06 PM
NO RATINGS
This whole episode was shameful.  I don't think you pointed it out in the article, but even if the software had left the throttle open, it's been demonstrated that the brakes can still stop the car with no trouble.  Apparently Toyota thought it was easier to settle and move on than to try to argue to juries that people had killed their families by confusing the gas and brake pedals.

Mike Galvin
User Rank
Rookie
Re: Brakes
Mike Galvin   3/9/2017 4:47:36 PM
NO RATINGS
I agree.  There was a good podcast on this from Malcom Gladwell (Revisionist History, episode called "Blame Game").  

Synder
User Rank
Rookie
Re: Brakes
Synder   3/10/2017 12:58:35 PM
NO RATINGS
The whole episode was shameful, but not necessarily the fault of the driver.  Those who still maintain that  a car can be braked to a stop from high speed with the engine at wide open throttle are deluding themselves.  Try it sometime, hold the accelerator fully open while traveling at 70 MPH and step on the brake pedal.  Do it sporadically as you might do when the car is acting strangely and will not stop. The brakes will fade pretty fast and although you may slow, soon the engine will overcome the fading brakes.  You absolutely will not be able to slow the car as quickly as if the engine were not at full throttle. 

The "simulations" that may have been done to prove that the brakes would work properly are just that, simulations, and are only as good as the assumptions.  For a good idea of the usefulness of simulations in real world emergencies, go watch the movie "Sully" and pay attention to the section where they talk about the simulations showing he should have been able to land the plane on one of two alternate airports instead of the Hudson River.

A failure analyst that had reoccurring problems with his Toyota fly-by-wire accelerator did some failure analysis and found tin (Sn) dendrite growth between the pins of the IC on the pedal that could tell the computer that the pedal was fully depressed when it was not depressed. A summary of this was published in EE Times, "Toyota accelerations revisited-hanigng on by a (tin) whisker -2012-01-10" 

The push-button start-stop switch requires the driver to depress the button for 3 seconds continuously to turn the engine off when it is in gear.  At 70 MPH a car will cover over 300 feet in the time required to turn off the engine.  That is a pretty scary scenario, expecting someone to hold that button down continuously for the time to cover the length of a football field in a panic situation.  I know that I likely would not do it.   

Toyota and other auto manufacturers are paying attention to this belated wakeup call to improve the software, firmware, and hardware of their cars. Autonomous vehicles will suffer the same challenges, no engineer or group of engineers, no matter how smart or wise, can anticipate all possible situations.

 

IanD
User Rank
Rookie
Re: Brakes
IanD   3/13/2017 12:36:28 PM
NO RATINGS
The suggestion that you can't stop a Toyota -- or almost any car -- with the accelerator pedal mashed to the floor is bullsh*t. A normal car like this can't pull more than about 0.3g under acceleration, but can decelerate at about 0.9g -- and there's enough braking power to lock the wheels, so some to spare.

Even without allowing for this excess it can decelerate at at least 0.6g with a wide-open throttle, and maybe as much as 0.9g if the brakes have enough spare power (which most have). In other words the worst that can happen if the throttle sticks open is extended stopping distance, the best is no increase at all.

No modern car will suffer from brake fade with a single stop even from maximum speed, where the energy dissipated is far higher than any of these so-called "runaway" accidents. If it did, it would be banned from sale as being unsafe.

All this has been proved using real cars in real tests, not simulations -- the worst case is that stopping distance increase slightly, none ever showed inability to stop or brake failure/fading.

In other words, if you press the brake pedal -- not the accelerator -- hard, the car will stop. Every time. No exceptions, even with a stuck throttle. Unless you pressed the wrong pedal...

EmbEngr17
User Rank
Rookie
Re: Brakes
EmbEngr17   3/13/2017 8:16:20 PM
NO RATINGS

"In other words, if you press the brake pedal -- not the accelerator -- hard, the car will stop. Every time. No exceptions, even with a stuck throttle. Unless you pressed the wrong pedal..."

This is simply not true.  NHTSA has data showing that it takes >100 pounds of force with brake vacuum depletion, which is more than a little old lady is likely to muster in a crisis.  For that matter the Saylor crash showed brakes burned out, which is inconsistent with your statement.

Consumer Reports has a video showing that an adult male has trouble stopping a car without pumping, and is unable to stop it after pumping brakes (which is a normal human reaction, especially for older drivers who trained before ABS).  Their scenario is stuck pedal or floor mat entrapment, but there is no reason to believe that is any different than a software defect that commands wide open throttle.

https://www.youtube.com/watch?v=VZZNR9O3xZM

There's my data that brakes don't overcome throttle in some situations. I'll be happy to take a look at any hard data you have to offer that applies to the Toyota vehicles in question.

IanD
User Rank
Rookie
Re: Brakes
IanD   3/14/2017 5:15:54 AM
NO RATINGS
OK, with the engine stopped and vacuum reservoir emptied by repeated brake applications after the engine stop, it's more difficult to stop the car. Congratulations, you found a theoretical corner case.

Now show me a single real-life case where this happened -- not a badly-researched press story, not a driver saying "I couldn't stop, my brakes were burned out", a verified post-crash forensic analysis showing that this really happened.

Have you ever wondered why all these cases only seem to occur in America?

ANTONY ANDERSON
User Rank
Author
Re: Brakes
ANTONY ANDERSON   3/17/2017 9:46:15 AM
"Have you ever wondered why all these cases seem to occur only in America?"

Sudden acceleration incidents are not confined to the USA. There have been a number of SA incidents in the UK, New Zealand, the Philippines, Malaysia, Australia, France, Sweden, United Arab Emirates, Zimbabwe, Korea.

 

Antony Anderson

Newcastle upon Tyne UK

 

IanD
User Rank
Rookie
Re: Brakes
IanD   3/17/2017 9:58:40 AM
NO RATINGS
OK, "most of...in America" -- happy now? Could this possibly be linked to the litigious society culture in the USA together with the number of lawyers? (more than the rest of the world put together IIRC)

dt_hayden
User Rank
Author
Re: Brakes
dt_hayden   3/15/2017 10:04:34 AM
NO RATINGS
Car & Driver found that at 70mph, the braking distance was increased slightly in full throttle versus closed throttle situations.  It was only at highly elevated speeds (100mph), involving a lot of kenetic energy to be dissapated, that some of the brake systems tested struggled or could not stop the vehicle before overheating.    Most "UA" events I have heard claims of are in low speed situations (i.e. parking) where brakes should be able to dominate the throttle no problem.   http://www.caranddriver.com/features/how-to-deal-with-unintended-acceleration

elizabethsimon
User Rank
Author
Re: Brakes
elizabethsimon   3/13/2017 2:32:18 PM
NO RATINGS
@ Snyder

The push-button start-stop switch requires the driver to depress the button for 3 seconds continuously to turn the engine off when it is in gear. ...

That is the dumbest interface ever... It's releated to the reason that I didn't buy a Toyota pickup. This whole concept that you can start and run the car without even finding the key to put it in the ignition was a bit creepy (apparently the dealer has universal keys that work for every car on the lot)

Every motorcycle that I've owned has a big red "engine off" switch that cuts the power to the engine. Of course they also have a clutch which works just as well to cut power to the wheel.

stixoffire
User Rank
Rookie
Re: Brakes
stixoffire   3/17/2017 5:19:59 PM
NO RATINGS
Depress the switch for 3 seconds when it is in gear .

This actually doe smake sense, it keeps someone from hitting that button accidentally - sometimes the someone can be a child, or in some peoples cases they have stuff in the front of the vehicle with them and it bumps the button. So yes I can understand the turning off by holding it in for 3 seconds.
Push button start has been around for a long time - Corvair..

Point is ergonomically a KEY makes much more sense than trying to engineer a pushbutton to do the same thing only less effectively. To your point.
If I want the engine off now - I want it off NOW, not 3 seconds from now.

jyavins
User Rank
Author
Re: Brakes
jyavins   3/10/2017 5:56:48 PM
NO RATINGS
I owned a Toyota when these incidents (there were several of them) were in the news. Regardless of any presumed computer failure, the drivers involved must have been pretty dumb. How hard is it to turn the ignition key? Toyotas (and most other cars) require that the key be pushed in to be moved from "accessory" to "lock", so there is little likelyhood of the steering wheel becoming locked. Moreover, power steering remains effective as long as the engine is turning over because it's in gear. Those too flummoxed to think of turning off the engine might at least shift into neutral.

Kevin Neilson
User Rank
Author
Re: Brakes
Kevin Neilson   3/10/2017 8:00:55 PM
NO RATINGS
I think they panicked and sensible solutions didn't occur to them.  In the heat of the moment they must've forgotten about the parking brake or gearshift.  

EmbEngr17
User Rank
Rookie
Re: Brakes
EmbEngr17   3/12/2017 8:11:25 PM
NO RATINGS
Not buying that for the Bookout Trial.  There were 150 feet of skid marks leading up to the crash site.

Kevin Neilson
User Rank
Author
Re: Brakes
Kevin Neilson   3/13/2017 5:26:28 PM
NO RATINGS
Bookout pulled on the parking brake.  The parking brake activates only the rear tires.  To me, this is not evidence of a bug, but just indicates that the car was dragging the rear tires while she simultaneously had her foot on the gas and hand on the parking brake.  This does not seem like a great mystery that can only be solved with cosmic rays and bit-flips.  Bookout was also geriatric, which is not proof of anything, but old age is highly correlated with pedal misapplication, according to the NHSTA.

Also, the 150' of skid marks is the sum of the lengths of the two skid marks: http://www.safetyresearch.net/blog/articles/toyota-electronics-guilty-bookout

stixoffire
User Rank
Rookie
Re: Brakes
stixoffire   3/17/2017 5:14:42 PM
NO RATINGS
Some point out that brakes were not pushed, transmissions were not disengaged or the gas pedal was mistaken for the brake.
1: Gas Pedal mistaken for the brake - a person is accelerating when they should not be - they want to hit the brakes but hit the gas. Logical Fault here - they were already in a condition of acceleration that should not have been occurring.
2: Not all drivers are consciously aware of how their car works or does not work, it is expected that if you are using software to control the vehicles acceleration or braking that it should work flawlessly, and have redundancy just in case.

I know that not all failures stack up and give a bright light hey I am the cause of this failure condition so that an engineer can track it down and declare it. In some cases an engineer doubts the condition entierely and utterly discards the possibility. If a technical person captures the moment in film or audio - the engineer is amazed and says that can't happen. And then upon research yeah yup, yes I see what this is, we had.. blah blah blah.

I myself have experienced sudden acceleration in aToyota Siena and the vehicle continued to accelerate. I was driving on a highway with cruise control, I needed to transfer from one highway to another so I tapped the brake to disengage the cruise control, I took the off ramp my feet are no where near the brakes or the gas because i do not need to traffic is extremely light. I get to the bottom of the on ramp and press the resume button on the cruise control. The vehicle accelerates very quickly 55, 60, 65, 70, 75 , 80, 85 - I am way past my cruise control setting at this point and expecting it to have levelled off at 75 from "overshoot" but it does not it is still accelerating when I am at almost 90 miles per hour I turn the cruise control off and press the brakes. So someone telling me they do not have an issue is like telling me the Earth , the Internet and people do not exist.



TonyTib
User Rank
Author
Re: Brakes - and mechanical breakdowns
TonyTib   3/20/2017 7:07:35 PM
NO RATINGS
Sure, unintended acceleration happens.  Sometimes it's the driver's fault; for example, I believe that the unintended acceleration that US Audi 5000 drivers experienced was due to driver error (pressing accelerator instead of brake)

However, the Ford Escape / Mazda Tribute had a recall because the throttle cable could stick and cause unintended acceleration -- and I believe this happened to co-worker of mine on I-280.

state-machine.com
User Rank
Author
How bad is bad enough?
state-machine.com   3/10/2017 8:41:44 PM
NO RATINGS
The title of this article asserts that the "Toyota's Firmware Was Not a Killer". This is a fundamentally faulty logic, because the author cannot prove that the firmware didn't, in fact, kill people.

Also, as I remember correctly, the experts working for the plaintiff did offer a theory that the fault happened due to a stack overflow, which corrupted the operating system variables that happened to be conspicuously just below the stack. The experts further showed that Toyota's stack use analysis was faulty and the stack could overflow. Finally, they also showed that once the bit corresponding to TaskX was flipped in a real vehicle, all other fail-safes didn't prevent the unintended acceleration.

But this article raises an interesting issue of how much proof is sufficient to implicate complex firmware like this. Please note that error rate of just one per many millions of hours of operation is sufficient to reproduce the observed fatal accident rate. Yet, the author demands that expert witnesses reproduce such incredibly intermittent event at will, effectively performing debugging of the system for the car manufacturer. All this while receiving no support or very reluctant support from the actual developers of the firmware.

So, the question is: How bad is bad enough for a complex firmware to be?

EmbEngr17
User Rank
Rookie
Re: How bad is bad enough?
EmbEngr17   3/12/2017 8:09:44 PM
NO RATINGS
This is easy: there are software safety standards that answer precisely that question. These days it is ISO 26262 for passenger vehicles.

Back in the late 1990s and through the 2000s it was MISRA Software Guidelines, which Toyota could have followed, but did not.  Note that these are much more comprehensive than just MISRA C. It is an entire software safety process of many hundreds of pages in length that outlines a SIL approach and acceptable practices for safety.

If you could follow the standard but you don't, and your vehicle kills people, that tells me that your software was bad enough to be a problem.

state-machine.com
User Rank
Author
Re: How bad is bad enough?
state-machine.com   3/12/2017 9:09:12 PM
NO RATINGS
My point exactly. Even as the Toyota firmware turned out to be a complete BBM (Big Ball of Mud), the author of the article assures us that it certainly was "Not a Killer". He bases this verdict on his second-hand analysis of a redacted deposition of an expert witnesses.

To implicate the firmware, the author accepts nothing short of reproducing the killing of the passengers at will.

The problem with this standard of proof is that highly intermittent problems that show up only once per millions of hours of operation are by definition difficult to reproduce in the lab, or in tests of a fleet of just a few dozen cars.

DavidMCummings
User Rank
Author
Re: How bad is bad enough?
DavidMCummings   3/18/2017 12:52:36 PM
NO RATINGS
Please see my response to your first comment. As I say there, the title was created by EETimes without my knowledge, and does not accurately reflect what I am saying in the article. Thank you.

dt_hayden
User Rank
Author
Re: How bad is bad enough?
dt_hayden   3/13/2017 3:17:25 PM
NO RATINGS
Regarding logic, the burden of proof is on the accuser.  You are not guilty of robbing a bank because you cannot prove you didn't rob the bank.  You are guilty because someone proved you did rob the bank.

In a case like this, it comes down to statistical probability.  it's not a good sign when the judge and attorneys are asking "what are statistics" and the reply is "something with numbers".  Having an expert witness who declares code "unsafe" for having 10,000 global variables shoots his credibility for me.  Sloppy perhaps, not necessarily unsafe.  Again the burden is on him to prove it is unsafe.

DavidMCummings
User Rank
Author
Re: How bad is bad enough?
DavidMCummings   3/20/2017 6:25:12 PM
NO RATINGS
I am the author of the original Embedded.com article. I just now discovered that this version of my article was posted by EETimes. While I am delighted that EETimes posted it, I was dismayed to see that they changed the title without my knowledge. The new title, "Why Toyota's Firmware Was Not a Killer," makes it sound like I am claiming that I have found that Toyota's firmware was not responsible for the accident. If you read the EETimes summary under the title, not to mention if you read the article itself, you will see that my point is that the plaintiffs' theory, which convinced the jury to find Toyota guilty, is not supported by the evidence and is not credible. Of course that is not the same as saying that I have determined conclusively that the firmware was not responsible, which is what the EETimes title implies. I encourage you to read the full peer-reviewed IEEE article to see all of the technical details on which my conclusions are based. In the meantime, I will see if I can get EETimes to change the title to one that more accurately reflects what I am really saying.

DavidMCummings
User Rank
Author
Re: How bad is bad enough?
DavidMCummings   3/24/2017 9:56:02 PM
NO RATINGS
Now that I've addressed your concern about the title, which turns out to be a non-issue, I'd like to address some of your other points that also reflect a misunderstanding of the facts. With respect to stack overflow, as I show in my IEEE article, the expert merely claimed to have determined that there was significantly less safety margin in the stack sizing than Toyota thought. He did not find any pathways through the code that overflowed the stack, and he did not observe a stack overflow in any testing. However, he represented to the jury that he had found an actual occurrence of stack overflow, even though he found no such occurrence.

With respect to the hypothetical flip of the bit corresponding to Task X, as I show in my IEEE article, if that were to have happened, the Brake Echo Check fail-safe would have prevented unintended acceleration once the driver stepped on the brake. That fail-safe executes on a separate processor from the main processor on which Task X executes. No evidence was presented at trial to show that this fail-safe would not have worked properly, and all of the expert's test results presented at trial showed it working properly. Your assertion that the expert "showed that once the bit corresponding to TaskX was flipped in a real vehicle, all other fail-safes didn't prevent the unintended acceleration" is simply not true.

Regarding your criticism that I based my assessment on a "second-hand analysis of a redacted deposition of an expert witness," that is not true. My assessment is based on the trial testimony and slides (not a deposition transcript) that the expert made available to the public, and which he then invited us to use in "judging for ourselves." So he clearly thought that his testimony and slides were sufficient to assess his technical arguments. Indeed they were, and they revealed that his technical arguments were seriously flawed and misleading, that he didn't show that it was more likely than not that the death of Task X caused the accident, and therefore, that the jury reached the wrong verdict. I encourage you to read the complete analysis in the IEEE article.

traneus
User Rank
Author
Documented example
traneus   3/11/2017 10:34:40 AM
NO RATINGS
See www.eetimes.com/document.asp?doc_id=1323903 for a documented example where the acceleration was not commanded, the brake was commanded (as shown by diagnostic readouts) and the car did not stop.

leftfootbraking.org
User Rank
Rookie
Why Toyota's Firmware Was Not a Killer.
leftfootbraking.org   3/15/2017 12:40:48 PM
NO RATINGS
The simplest explanations are usually closest to the truth but also froth with politics. Mark Saylor and most of the other Toyota drivers committed a Type 2 Right Foot Pedal Error, part of the Right Foot Braking Epidemic. Neither the drivers nor Toyota were to blame but instead the Real men who are responsible for at least 7 deaths each day and go unpunished. What happened to the investigative journalists who should be investigating and exposing this mess?

THE BLAME GAME – REAL MEN AND THE TRAGEDY OF RIGHT FOOT BRAKING ON AUTOMATIC CARS.

The car has always been one of the most dangerous modes of transportation from not only the driver's point of view but also other victims such as cyclists, pedestrians and people including children, inside supposedly safe buildings.

There are times when car crashes reach epidemic proportions and at this point politicians and governments play the blame game. The favorite and easy targets are the automobile manufacturers and the drivers. The governments in charge of driver legislation and training never blame themselves so if you don't mind we will.

This is the story of how things went horribly wrong because testosterone got in the way. You see, Real men in or out of government believe that every driver should brake an automatic car with their right foot. Truth be told Real men would like us to all drive manual transmission cars but the women got in the way and ruined a manly tradition.

We now have evidence (If the Real men would look at it), that indicates that right foot braking on automatic cars is:

1. The cause of car-building/parking lot crashes.

2. The cause or the weakness (poor stopping distance) of car-cyclist, pedestrian, etc crashes.

The politicians will argue that this tragedy has been studied. That is true, reports DOT HS811 597 and DOT HS812 058 have been issued. The reports say:

1. A lot of people are being killed or injured by right foot pedal error. The forecast is that between 16,000 and 18,000 car-building crashes will occur in North America this year with an estimated death rate of approximately 600-1000 many of them children. This means since the automatic transmission became popular over 30,000 people have been killed, millions suffered life changing injures and the costs have been in the billions.  AND THIS IS ONLY IN PARKING LOTS. If one does a little math and estimates the effect of including the cars on roadways, 150,000 deaths, millions and millions of injuries, billions and billions of costs is a reasonable estimate!

Actually the boys who wrote the reports couldn't bring themselves to use the phrase" right foot pedal error". They called it the politically correct phrase "pedal misapplication". Oh and by the way they said it was mostly the fault of the women and old people drivers. I'm quite sure this met with the approval of the Real men who are convinced that neither of these two groups should be allowed to be on the road.

2. It appears that there isn't any car that can be designed for all the different sized drivers that will prevent right foot pedal error so we will just have to do the best we can.

Did it not occur to the many learned professional people who wrote these reports that just maybe they should have concentrated on the driver - not the car. That maybe there was a braking METHOD that would allow all drivers of all shapes and sizes, to for example stop a car in a distance 30 – 40 feet shorter than they could if they used their right foot to brake.  And that the use of such a method would eliminate right foot pedal error? Could they have had that kind of tunnel vision? Or were they told not to go there? After all who is going to stand up and apologize to the 30,000 – 150,000 people who have lost their lives.

To further rub salt in the wounds, the Real men in charge of driver legislation and training continue to hold right foot braking on automatic cars up as the gold standard and constantly deride all other methods of braking such as the Left Foot Braking Method, leftfootbraking.org. This with the full knowledge that they have Never Had Any Scientific Data That Proves The  Superiority Of Right Foot Braking On Automatic Cars!

 Their science can be summed up by quoting a government transportation expert who said, "That's how it's always been taught"! These are the people you are trusting to teach your children how to stay alive while driving but as pointed out on the website, have been set up to FAIL!

There is an answer to this tragedy and you can help:

1. Go to leftfootbraking.org and learn about how the Real men should have taught drivers to brake an automatic car. It is the Left Foot Braking Method and it is not just trying to brake with your left foot so read carefully.

2. Contact your government and demand they support and run a scientific study comparing right foot braking on automatic cars with the Left Foot Braking Method.

3. We are confident what the results will be. So based on the results of the scientific study, we then want you to demand the governments BAN the teaching of right foot braking on automatic cars.

4. And finally we want you to demand the Real men stand up and apologize to the 150,000 dead people and the millions and millions who have had life changing injuries.

Oh and by the way the semi-autonomous car is not going to help. Once the thrill is gone, the driver switches back to his normal "I am in control" mode and the possibility of right foot pedal error. We have already had our first SA end up in a building!

Trevor W. Frith

Was it driver error or the way we taught them to brake?

leftfootbraking.org

Trevor W. Frith is the webmaster of leftfootbraking .org and is calling on those in charge of driver legislation and training to commit to a scientific study comparing right foot braking on automatic transmission cars with the Left Foot Braking Method. Mr. Frith may be reached at leftfootbraking2014@gmail.com,

For further verification search; 'Car Crashes into Building'-The All-Too-Common Headline. Ameriprise Auto and Home Insurance and download podcast Revisionist History by Malcolm Gladwell, Episode 8, Blame Game. We also worked with Richard Schmidt before he died.

 

 

 

 

 

 

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)
Like Us on Facebook
EE Times on Twitter
EE Times Twitter Feed