It's been a hoot watchin' Toyota scramble to settle lawsuits and the federal criminal probe out of court following Michael Barr's findings regarding bugs in the Recall King's electronic throttle control software. The government's well-orchestrated effort at a "Michael Barr news blackout" isn't working quite as well as it hoped it would, thanks to bloggers, whistleblowers, and Mr. Barr's peers publicizing his findings in trade journals and at engineering conferences.
Out of the billions of dollars worth of Toyota's payouts, this week's $1.2 billion wortha payola to end the federal criminal probe is perhaps the most notable. Not that the Recall King is worried. As Toyota's corporate-controlled government friends well know, the Recall King has a cash stash of 60 BILLION BUCKS and isn't the least bit worried about a few billion forked over as part of an effort to shut things up about wire fraud, lying to Congress, and Michael Barr's findings.
Thanks, EE Times, for having Michael Barr as the featured guest at your upcoming conference. Right on!!!
I read the court testimony and one part of his testimony stands out as outright lying.
He calims the EDR is tricked by the ECU into not recording the brake state correctly but it is pretty obvious that the brake switch (on/off) is not sampled by the engine control unit for the EDR system as the sampling rates are different (the EDR samples the brake switch immediately according to reports and takes ms to get updates from the ECU which is not likely to mess up a switch state which is probably electrically tied into multiple controllers as it isn't used for anything other than safety critical functions not to mention task X/K isn't reading brake state as that is another CPU's job which also is tasked with transmitting the data to each controller so I don't know what BS he is talking about in that transcript)
He presumes a number of things in contrary to a lot of real world testing which doesn't support his claims and appears to bend the truth in a number of situations. I would like to see the actual report before praising his expertise. Without independent verification basing an entire case off one expert when many others have opposing conclusions is not very "scientific" as he likes to say.
Using a debugger to induce deadlock is not simulating a stack overflow. (Which would likely wipe out everything causing the WDT to reset) Having guard bands would require more CPU time to poll them constantly and having explict memory managment requires a beefier OS and CPU. His mention of global use on an embedded system and the fact we don't know what brake force he used in testing leads to questionable quality for his testimony. (Not to mention he was hired to say Toyota is at fault while a number of independent groups have found no fault).
And this article is just an ad for a conference which he/and this site are promoting which just reeks of conflict of interest and flooding the internet with his supposed claims when on programming sites his "advice" is arugable at best and not applicable to all embedded controllers but is advertised as golden rules.
EDR reports and real world testing conflicting with his unpublished report makes me think he is just hired to say what he is saying. (His book are also not really books, one is a dictionary, another is a handbook at best, and the last is actually out of date)
Writing complex failsafe code is never going to be "failsafe" it is ignorant to think bugs can't ever exist in code. Sure program to reduce them and try as hard as possible to not but keeping some extremely lightweight and if possible not even software based failsafes like (oversizing brakes, power off, allowing shift to neutral, individual battery cell monitors, independent systems, etc... will mitigate defects)
Look at Boeing's 787 battery fire where it is a clear battery monitoring circuit failure in a dual channel redudnant controller which failed to inhibit 45A charging when it was clear that one cell failed.
In toyota's case almost all incidents can be directly attributed to driver error and a few to mechanical placment of aftermarket floormats. The supposed bugs will always exist and it is impossible to make code bug free. Thinking some coding standards, principles, and regulations will ensure safe code is dangerous thinking and will later be used as an excuse to remove hardware and multi-system thinking in fail safe design.
A good system should be able to have crappy code and total messes while still being safe. In such a case with average quality code the number of incidents will be very low if at all (toyota's case).
Also his slides contain a fatal flaw in reasoning. In an UA event assuming the driver has their foot on the accelerator (required for the throttle to be stuck) or uses cruise control (no foot on either pedal) then requiring that the brake pedal be not depressed before allowing the fail safe to kick in makes sense as there are ligitamate uses of dual pedal driving (if you know how to drive, which most don't and if you don't know why you would use two pedals then look up steep hill driving and stop riding on your transmission). If the driver is assumed to not be pressing the wrong pedal then only the acceclerator should have been pressed. All in all even with the "bug" the probablity of it happening is low and not demonstrated to be the cause of the acceleration in that case.
In addition his own graph makes clearly demonstrates before the failsafe which does work kicked in the car still slowed down just as much as it accelerated even after pumping the brakes which was an obvious attempt at reducing the braking speed which would have been much more commanding if he just pressed the brakes directly. In addition pumping the brakes involves removing the pedal and pressing down again so the fail safe would work. Making it cut off engine power at any braking level is dangerous and may cause loss of engine power during driving which could cause far more accidents (GM's case).
His fatal flaw single bit-shift isn't in the code and I find all his conjecture dubious without the actual report. He is killing tasks 4,5,1 by inducing a deadlock in all likelyhood. By using a debugger he is also bypassing operating system protections and likely used boundry scan to alter a very special OS memory location which is highly unlikely to be modified due to OS memory mapping. All of his talks about seg faults and other bugs are unrelated to the fact the single bit he flipped is likely part of task contention and isn't accessible to anything but the OS. Nor does he talk in detail about how a stack overflow would have killed everything if it impinged on the OS memory space as it would have smashed far more than one bit value. (By using a debugger he doesn't demonstrate the supposed cause just that you can crash the car with a debugger which is obvious)
If they catch it in practice or add monitor code to a custom test rig and run it in simulation to prove it then I will be convinced. But without any actual real world testing results (where are the conflicting EDR records, please link them in if you have them as I can't seem to find them) finding bugs doesn't mean they happen. (Sure not ideal but no code is bug free)
Compared to the FADEC and Boeing 787 battery stuff I've read reports about Toyota is probably a better programmer than the contracted out work done for aviation grade stuff so I don't really know what the fuss is all about.
Considering eetime's has an article saying how Boeing's battery "fix" wasn't software (WHEN THEY DIDN'T FIX ANYTHING, OR EVEN KNOW WHAT WENT WRONG) and the battery still fails just inside a box and it is a critical backup component. (Lithum batteries have safety critical firmware too and the published report clearly stated it didn't open while the charger likely lit the whole thing on fire with 45A in) With 200+ passengers I don't want my battery in a box when in a birdstrike situation which does happen you really need those batteries to work.
I came here just a week ago but I don't think I'm ever going to read this site anymore. The consequences of a plane crash are far larger than a car and I'm sorry to say that mfg's are not recieving a proportionate amount of coverage/flak for their failings.
Drivers have ample workaround even if a ECU is total and demonstrated garbage but a plane doesn't if the APU fails to start or backup ESS buses fail all that happens is reductions and loss of function which in an emergency it get deadly in a car stopping typically won't result in you falling out of the sky something plane's don't have the luxury of being on the ground in most oprating situations.
"zewde," it's incredibly "fascinating" that Toyota could keep things quiet for so long, and that consumers were forced to wage full blown lawsuits before the facts came out. The public now has every right in the world to question the relationship between Toyota and NHTSA, especially with Toyota whistleblower Betsy Benjaminson making some pretty strong comments about the issue. Her remarks tend to be supported in yesterday's LA Times article:
"In perhaps the most glaring case, Toyota staffed up with former NHTSA officials as it faced an inquiry into sudden unintended acceleration in its Toyota and Lexus vehicles. Over 10 years, more motorists died from such accidents in Toyota and Lexus vehicles than in cars from allother manufacturers combined." (Emphasis mine)
Sounds like Toyota's cars are already "self-driving."
The Toyota case is fascinating. This could be an opportunity for some real insight not only into what happened but what the issues are going to be moving forward for the automotive industry as it heads toward the future of the self-driing car.
Since there's no evidence that Toyota has corrected the problems Mr. Barr discovered, it might be kinda tough to get someone from the Recall King to explain how they did. Cases of sudden unintended acceleration continue to pop up, and in one of the latest, cops have ruled out driver error. Meanwhile, the founder of the law firm that won the case in Oklahoma has produced a video wherein he accuses Toyota of a cover-up: http://www.youtube.com/watch?v=LE7Xxs3g4yU
Maybe someone from Toyota would like to explain how they manage to deceive the public.
That would be a very interesting panel to have Toyota and Medtronics (and other companies) talking about how they dealt with their particular software flaws and legal issues arising from the flaws, but I fear because of the litigation, no company could attend.
NASA's Orion Flight Software Production Systems Manager Darrel G. Raines joins Planet Analog Editor Steve Taranovich and Embedded.com Editor Max Maxfield to talk about embedded flight software used in Orion Spacecraft, part of NASA's Mars mission. Live radio show and live chat. Get your questions ready.
Brought to you by