Why Did It Take GM a Decade to Identify Ignition Switch Problems?
GM says it became aware of issues on its vehicles as early as 2004, when it received a field report of Chevy Cobalt vehicle losing power after a key moved out of the “run” position, according to documents filed with NHTSA. Cobalts in the 2005-2007 model years have been recalled.
"I see no way around the apparent fact that Toyota's Task X can crash, fail to reboot, and cause the throttle signal to show full open."
Showing that something is possible is not the same as proving that it actually happened. What is the probability of this "apparent fact" happening under normal driving conditions? As this is a technical matter, the probability of its occurrence should be able to be determined by simulation or actual testing. Has anyone provided such data?
"I agree that under such circumstances ... a totally alert driver *should* be able to stop the car."
The drivers in all of these cases claim to have been fully alert and pressing fully down on the brake pedal.
FADEC (basically a jet engine equivlent to an ECU) with horrible programming to the point where even beta testing showed it to be totally unfit for use and cause total engine destruction and possibly many other failures. The catagory 1 defects in the program where likely totally obvious and known oversights which just where not fixed. Having code which looks bad and an outside "expert" claims in unmaintanable is not the same thing as a clear software fault that even in production without any alterations or debuggers which causes an engine problems in testing being called unintelligable and using/relying on undocumented hardware features.
(Arugably having a single main CPU and smaller monitor CPU would have been better for that programmer as the monitor CPU has very simple tasks to carry out and even if they totally botch the main CPU task X they are not running two copies of it or don't have to try and program two different giant task X's but one large one and a tiny task Y) The programmers could be a failure point as well and a pretty catastrophic one in that helicopter's case.
In that case the FADEC programmers were making code so horrible it was using features that where not even supposed to be used on the CPU so they were in essense making the hardware run out of spec which is something that Toyota hasn't appeared to do from the testimony even as that would be a pretty glaring safety failure if they did do that but they used two different CPUs and appear to have sound safety design for a ground car application. Code review in the helicopter case said it was totally unfit and when pushed into service the helicopter crashed and the software was changed and the out of spec CPU as well and they went on. Toyota hasn't recalled and no actually documented cases have been reported so no catastrophic software fault is assumed to exist.
GM has a known failure with a known fix with a long time delay and a non-recall warning to dealers which only now is being addressed years after the cars went out of production and under new managment. GM had a total systemic engineering to managment failure while Toyota had a PR disaster which was technically not their own doing (Maybe they could have wording things better than saying those dead drivers basically killed themselves for the most part but techically they are right in the bulk of the cases). (GM on the other hand has people dying and crashing not because of driver error but a simple mechanical problem with nothing less than a mechanical switch dedent being under spec)
Task X can fail if you make it fail but that doesn't really mean anything nor do they even say what bit they are flipping is it a debug value or something what is it we don't know and can't really judge how fragile task X is as it is like saying a debugger can pause/breakpoint the CPU (That better not be the bit they were flipping). Sure maybe radiation drops the CPU into debug mode and simulates a fake breakpoint and kills task X while keeping the watchdog timer happy. Conjecture yes, proof none, same as redacted testimony)
Rebooting a ECU while the engine is on is dangerous and it isn't just a PC computer running an engine. Shutting of the entire ECU would cut off sensor data to other independent controllers and could cause airbags to fail to deploy or a host of other issues with systems which expect the ECU to be running when they are up. Which would be more dangerous than 100% throttle which doesn't even happen in practice. Want to reboot a FADEC on a jet engine (Don't do that, don't even try plugging a debugger into a running FADEC on a real jet engine) Sure those are far more reliable and robust but rebooting isn't a good solution. Memory scan/encryption/error correcting codes/... (They did have memory error correction I think but a debugger would be a valid memory change) might do better. Obviously having N+1 redudancy would be safer but that would have added costs and how much additional redudancy and diverse software do you want and what does that cost.
In GM's case it is the exact opposite even functionally the car stops running when it should be running and the user doesn't want to stop or power down the car and in Toyota's case it is the driver pushing the wrong pedal.
GM doesn't have a control system fault they have a mechanical switch design fail that they knew about and didn't correct for 10 years.
Toyota has no fault and paid 1.6 Billion in "economic loss" suits and undisclosed amounts to other cases.
GM recalled 1+ million cars to replace said key part after the cars went out of production.
Toyota hasn't recalled the ECU in question.
GM the user should turn the car back on.
Toyota the user should stop pressing the accelerator pedal, or turn the car off, or press the brakes, or shift to neutral, or... (GM has one problem the switch, Toyota has a human factors issue)
GM didn't change the part their supplier did and they never recalled the switches even though people where dying in crashes where it was directly attributed due to the EDR reports.
Toyota has no part change not even a silent one, no recall, no deaths attributed to the ECU, no EDR reports, nothing just a lot of "pedal misapplication" and mechanical problems.
GM has people running a big overly complex task X which failed silently for ten years.
Toyota has task X still running its giant blob of code today without any issue. Blob code doesn't mean it is going to kill you. Might not be pretty or maintainable but it seems to work and no amount of analysis can prove/disprove a system is bug free or not.
GM has a single point of failure in a mechanical part.
Toyota has no recorded point of failure in the ECU and many mitigations even if it did occur.
There is no way to make an ECU bug free but that doesn't mean it will crash you car if task X stops. A properly designed ECU (Which toyota's does appear to be by history and use) can still function safely even without a kitchen sink task X.
I don't give them a break but I don't expect them to write bug free code (impossible) nor have excessive amount of redudancy in one aspect of the car (ECU).
GM on the other hand failed in so many areas in managment, engineering, supply chain control, recall, ...
I see no way around the apparent fact that Toyota's Task X can crash, fail to reboot, and cause the throttle signal to show full open.
I agree that under such circumstances, same as in the GM case, a totally alert driver *should* be able to stop the car. In both cases, the steering remains functional, and the brakes too, sort of. The problem is that when you suddenly present a driver with a situation he's never experienced, very frequently the driver won't have the correct and quick reflexes necessary.
The important point being, the driver needs to do the SAME THING, in the Toyota and the GM control system faults. Push really hard on those brakes. (Assuming you're right that the Toyota case doesn't totally incapacitate the brakes unless pumped once, as was demonstrated.)
GM's excuse was that they had already changed the part, although without a change in the P/N, so they thought the problem was solved but couldn't properly track it.
I would be amazed if Toyota doesn't update this Task X to absolutely avoid the possibility that its crashing would not affect the throttle in any possible way.
Task X is part of the main CPU which doesn't have the A/D converter in it and the A/D task isn't part of the main boat load process as they said async tasks occured outside of it (interrupts anyone which likely service data coming from the monitor CPU containing the digitized sensor information)
The EDR is seperate from the ECU but would take the A/D readings from the monitor CPU and likely just sits there watching the digital sensor lines for the data between the monitor and sub CPU. It records both the pedal and throttle value and only the monitor CPU reads those and both check the values for sanity. The EDR doesn't care and would just write down whatever is going on. If the Task X is the A/D converter then I don't see how an A/D converter can be in the main CPU as it isn't in the publicly released diagrams.
The brake pump once thing is also patently false based on a number of public tests showing the opposite result with constant braking stopping a car even with full open throttle. (Yes lightly pressing the brake won't stop a car but that isn't exactly a revelation)
Also ignoring any unproven conjecture based "expert evidence" using a debugger to cause a failure just using the slamming on both pedals would stop a Camery faster than a Ford Taurus could from 70 MPH even with a depleted reserve you could still brake and slow down to a safe speed and stop if you know what your doing.
GM knew about the problem and had a fix but didn't do anything. At most assuming your right which I don't believe you are they didn't know of a problem and haven't needed to fix it ten year later so I don't know how they are similar.
Toyota never recalled the ECU software and Task X is still going strong and UA events are down to expected background levels and are almost always "pedal misapplication". So unless Toyota is killing people who try to talk I don't think there is an issue with the software. Maybe it isn't well designed and could do with some cleanup but it isn't shown to be defective but I can't prove that to 100% and neither can NASA but we can say within a wide margin that it isn't a likely cause by a long shot.
The problem with your thought on software is that it is virtually impossible to eliminate bugs from even safety critical software. Designing the software and hardware such that you can be as lazy as possible in the main CPU and lazy in the subCPU and have electrical/mechanical failures without causing death/injury is good design. Relying on good code relies on a select number of human programmers to be really good at what they do and that is a point of failure that could easily occur. Writing different pieces of software, getting electrical designers to chip in, and adding a mechanical engineer to design the brake/engine power balance makes it so that no one design failure could cause a problem.
In Toyota's case there doesn't appear to be any problem with any level of design short of jamming a debugger and commanding what is probably a throttle override routine which the fail-safe's are probably disabled when a debugger is attached as the ECU would be in a test-jig/pre-production mode where full throttle testing would be desirable and attempting to override it by physical inputs is unexpected and admirable that the car could even stop with a debugger bit flipping a select [redacted] value. This entire previous paragraph is conjecture because they don't explain things clearly and no sound technical details where given.
For a Toyota car to crash the user has to one press on the wrong pedal (or something that never has been recorded in practice happens and it looks like full pedal) and two not know what neutral means, not know how to use brakes properly, and not know how to turn off a car. (GM's got that in spades I guess)
Toyota is basically a human factors case plain and simple, GM's is a total design and subsequent handling failure.
And obviously I'm not saying write horrible code as that will cause failures but it is going to happen (doubtly it caused it in Toyota's case) and other design aspect can mitigate it and did in Toyota's just the user's didn't know any of the layers of mitigation steps they could have done. Even in the unfortunate improper footmat speed/firey crash they could have just crashed into the median to stop themselves and that would have been far better than flying into a ravine (Hindsight 20/20 yes but maybe some better driver training on car basics and emergencies would help, I read a story where an engineer used a controlled car crash to help another person, search "Driver thanks man who hit him on purpose")
Training 911 and drivers on what to do would be far more effective and reliable than adding quad redudant/diverse os/physically seperated ECU control. Sometimes the solution/problem isn't electrical.
And again obviously if they knew about a bug they should fix it but unlike GM's case where they did know Toyota doesn't seem to think there is an issue and is there a multi-million car recall to reflash the firmware or swap the ECUs (no there isn't) only ABS software bugs in suboptimal breaking distances (Old ABS software is far worse arugably), and mechanical issues where addressed.
Finally the biggest nail in the unintended acceleration cases is that pressing the brakes and throttle does stop the car and numerous news agencies tested it even without brake override. (Some claiming it didn't but their claims are dobious at best)
Didn't Task X death also affect EDR recordings? I think that came out as well. I think the worst aspect was that applying the brakes would not always work, unless they were pumped once. To me, that's the worst effect. And after that, the lack of power assist at full throttle, of course.
In the GM case, where the ignition went off, the brakes would instead continue to work, but without power assist. The biggest issue with the GM case is that the air bags wouldn't deploy when the ignition is off.
So in my view, both companies have done the same thing, including waiting a decade or so before addressing their problems. Although it's POSSIBLE that Toyota really didn't know they had a problem (hard to believe, when you read the transcripts, but I suppose the engineers in that program may have gone on to other programs and not obsessed over past work).
I'll agree that there was a lot redacted, so we aren't getting the complete story. Still, I think that what happens after the case resumes will tell more of the story. If I were the software designer, and a flaw was discovered in my software, even if the risk is very low, I'd certainly want to change it!
Oh yeah. Now that I think of it, I've that happen to me too.
Also if you can't read the little side bar on the side to tell that I'm not a EE|times reader you seem to be mistaken that I would have read this site's coverage.
I guess you care more about transcripts than engineering so its a bit strange. No actual evidence was presented in those very transcripts you provided. Unlike the GM case which has deaths with strange EDR sensor values the Toyota cases have normal EDR reports with strange user values of the dual redudant pedal sensors), even if task X fails and subCPU fails as the sensor values would either be contridictory or non-sensical but nothing of that nature has surfaced and they are just grasping at millions of dollars if they can't prove that it actually happens in practice. I don't want to go blaming drivers for their own deaths/injuries but it is unfortunate that people couldn't just admit they pressed the wrong pedal but human factors is much harder/variable than electrical/mechanical design.
I'm open to accept real hard non-redacted proof but a debugger causing a failure isn't a real failure and not a single EDR I know of showed it happened in an accident which in GM's case many showed strange contridictory sensor information.
Yeah, using a specialized software debugger to cause an UA isn't exactly proof of anything. Similar to the other "experiement" by removing the ECU and overriding the engine control electrically just using a debugger to alter the ECU in software isn't proof of anything as in debug mode the ECU won't function the same as in production as debuggers induce their own effects especially on installed hardware controlling a real time system.
Also they demonstrated their lack of expert knowledge by the very thing I mentioned before, "Does it make sense that you're braking but you are having to fight the throttle because it is open 50 " (Hills that's the answer, steep hills, you want the two to fight but I guess jerkly moving up the hill or just loading that transmission up is better I guess, doesn't matter all evidence is on the drivers fault for the most part) A debugger induced UA isn't proof and so much [redacted] it doesn't say much at all other than its their fault software fail no reason other than big software fail.
They didn't mention how the sub-CPU was bypassed or why it wasn't involved or what the debugger does to the ECU when connected and how it might affect results. Nor does there have to be a DTC to show strange data in the EDR. (Which records all the sensor values so unless you have a transcript of them showning an EDR report it never happened and all the EDR's pulled from accidents showed "pedal misapplication" as the primary cause)
Join our online Radio Show on Friday 11th July starting at 2:00pm Eastern, when EETimes editor of all things fun and interesting, Max Maxfield, and embedded systems expert, Jack Ganssle, will debate as to just what is, and is not, and embedded system.