The expert witness aptly describes the Task X as "kitchen-sink" task. It is designed to do just so many thing. So what happens when the Task X does? So many things could go wrong, and one of which is a loss of throttle control. Talk about a bad design.
It is true that the throttle function will be affected by the software malfunction, but why the throttle function is not being tested again in the worst case scenario again to justify the findings after Toyota Trial?
What caused the so-called "stuck pedal" wasn't the issue in this case. At issue was the software controlling the electronic throttle control system.
As the expert witness explained, the software in electronic throttle control is responsible for performing the sparking and the throttle control.
But there is another part of the software that is looking at the driver controls-- looking at the accelerator pedal and cruise control. So there is a part of the software looking at what the accelerator pedal position is, is it down, is it up, how much down. Then that is translating that into a calculatedthrottle angle.
That malfunction was the crux of the issue that was argued in this trial.
Actually, what this throttle position algorithm does is translate the pedal position (which is apparently determined by an unregulated analog voltage, corrected by the program, according to a separate article) into fuel and air delivery to the fuel injection system. When the car is not in cruise control. In cruise control, presumably the throttle angle is not examined at all, and the fuel/air command is supplied as a function of vehicle speed vs requested speed.
Worrisomely, brake application did not override these control signals if that control algorithm app died. *That's* the crux of the issue here, I think.
I have to agree that monitoring functions, especially in safety-critical systems, should be done independent of the control functions. A totally separate loop, software and also hardware.
Actually, a floored brake pedal, as claimed in this case, will override the engine completely no matter what tasks A, B, C, ... X, Y, and Z are telling the engine to do. The engine can wail away at full throttle, perhaps burning out the transmission, but the brakes will stop the car!
But I guess a decision was made that the story is much more interesting with a rogue "Task X" lurking in the engine control software.
"In cruise control, presumably the throttle angle is not examined at all, and the fuel/air command is supplied as a function of vehicle speed vs requested speed."
I think that what you may have meant to say was that the accelerator pedal signal was not examined when in the cruise control mode.
I understand that in cruise control there is still an inner throttle position control loop and an outer speed control loop is added. In effect the driver input via the accelerator pedal is disabled or ignored.
The outer control loop is a speed control loop - where a speed signal is fed back and compared with a set speed (speed reference) to give a speed error. Presumably it is the speed error that is fed in as a torque request to either speed up the vehicle or slow it down to match the actual speed to the set speed.
"I think that what you may have meant to say was that the accelerator pedal signal was not examined when in the cruise control mode."
Indeed. The angle of the accelerator pedal. Sorry for the ambiguity.
In at least some older cruise control systems, perhaps also on some new ones (I certainly haven't done any study on this), the cruise control system actually moved the accelerator pedal. So that the same linkage between accelerator pedal and carburator was used in cruise control mode, to maintain a constant speed.
Actually, in every cruise control system I've used, if the desired speed (set by the accelerator pedal position) exceeds the current CC set speed, the system will still throttle up; when the pedal is released, it smoothly returns to the set speed. So when things are working, the pedal is not ignored. If task X failed, you'd notice that you couldn't speed up, either.
Thanks Junko for the thorough coverage. I've learned a lot about the case.
There seems to be serious design flaw. In order to avoid any serious issue in any software system, the design shall always avoid deadlock. There shall always be a simple task to monitor the health of the system. A watchdog to reboot the system in case of deadlock is an avoidance mechanism; system engineer shall not rely on it.
To be honest, I'm quite surprise to read the report. Toyota is a very good company. They should know better. I wonder whether there is anything missing n the findings.
Nonetheless, Toyota will learn from it and make themselves better.
I myself have learned a great deal in following the Oklahoma case. The thing is, though, that this is not the end of the Toyota's unintended acceleration trial.
Toyota is facing another trial early Nov. -- this one will be in federal court in Santa Ana, Calif.
In many of the death and injury lawsuits, including Bookout's, plaintiffs claim that loose floor mats and sticky pedals don't explain all episodes of sudden acceleration and that the electronic throttle control system is at fault.
The reason why EE Times is following the case so closely is that the Oklahoma trial was the first instance when any of the testimonies by expert witnesses focused on software and hardware issues -- outside the floormat and sticky pedals -- became publicly available. Until now, such reports and testimonies have been sealed under the court order.
And one more disturbing fact. Bookout's vehicle, a 2005 Camry, wasn't included in the Toyota's recalls.
There have been many-many posts here about how the braking system should always be able to override the engine.
What about the anti-lock braking system?
Virtually every car has them and the control computer has the ability to release the brakes at any time depending on factors like invididual wheel rotation speed and so on. I don't know how the ABS is tied into "Task-X" but if they all use the same microprcessor, it's entirely possible the ABS will be affected too.
Thus, pushing on the breaks would have no effect if the ABS has released them, falsely thinking the car was in a skid condition. This seems to correlate closely to what some drivers have reported; that the brakes had no effect.
It seems there should always be a mechanical overide for emergencies like these. The parking brake, otherwise known as the "Emergency Brake" which it isn't, applys only the back brakes. And the actual brake pads are tiny compared to the front pads. It would be of no use in an engine runnaway situation.
I'd really like to know how the ABS ties into all of this.
Some very competent people have reported that the brakes failed to respond when they tried to stop the car from an unintended acceleration incident. Most people investigating this passed it off as "driver error". I understand this is a "panic" situation and people can get confused. But these people swear they were stepping on the brake and not the accelerator.
If the drive computer went into some strange runnaway mode where it caused the engine to accelerate, it also could cause the anti-lock braking system to release the brakes. The anti-lock brake system takes precidence over the brake pedal in a skid condition, as it should do under normal conditions. But in these cases there would be no way to stop the car.
This is all speculation and guess-work of course, but it seems to follow closely to the reports of the actual drivers who experienced it. Very intriguing report. And it gets more interesting all the time.
Exactly - as I said before, there are millions of vehicles on the road with this defective software. The loss of control condition is not occurring very often or we would be seeing a lot of Camrys in the ditch or being hauled to the scrapyard.
Still, it CAN happen - 'under what conditions?' is, perhaps, a question that cannot be answered. And maybe that points to the core of the issue - the software that controls safety-critical systems must be deterministic, that is, it must do action Z in case Y in time t +/- tx wher tx << t. Clearly the Toyota engine control software does not conform to this requirement. Why are we, as a society, letting Toyota off the hook here? Because it doesn't happen very often? I would suggest that it has happened more often that the published data imply - has every single vehicle/single driver fatal accident involving a Toyota been throughly investigated? Or are many of these written off as 'driver lost control of vehicle?' We are dealing with lucky survivors tales here rather than unequivocal data - and burying victims of a massive fraud.
It seems to me that Mr. Barr's work represents that unequivocal data - this CAN happen and, as engineers, we all know that what CAN happen WILL happen sooner or later.
Absolutely, @sixscrews! Dr. Antony Anderson's paper seems to address the NHTSA's faulty argument quite well. We, the general public and Toyota owners in particular, should not be subjected to any known risk. The public needs to have access to all the information.
This is not the same as the ABS system. ABS systems are designed to keep the wheels rolling because the coefficient of static friction is higher than that of sliding friction, and rolling front wheels can be steered while rolling back wheels will maintain control and follow the front wheels. ABS systems do not release the brakes to the point stopping distances are increased.
Unless you have some evidence that the ABS systems were compromised/defective such that the brakes would have been substantially released when the drivers claimed they had the brake pedals floored, it is irresponsibel to fail to mention that the brakes will stop the car regardless of whether the engine is at full throttle.
It is also irresponsible to fail to mention details of all the other "sudden acceleration" cases that have been investigated over the years and found to be driver error. There have been hundreds and perhaps thousands of other drivers in all makes and models of cars who swore their foot was on the brake, meanwhile all the evidence showed their foot was on the gas.
It would be good to also post the transcript of the Denso Monitor CPU code -- to see why it might also have potentially contributed -- Also most ECU /ABS code is supposed to also meet a set of MISRA safety checks as part of a Static Analyis -- It would be good to hear about this in the trial -- Additionally it might be good to see how any hardware features came into play.
@MS243, we wish. Denso's CPU was examined by experts. But all we are working with here is trial transcript; none of the reports or slides supplied by witnesses during the trial is publicly available at this point.
The trial, transcript and these discussions indicate that there are millions of vehicles on the road today with a potentially lethal defect. Toyota has already settled with the NHTSA and has that settlement to wave in any Camry owner's face (provided they did nothing and accepted the settlement terms). Am I correct about this? And, if I am, what is the next step? I own a 2004 Camry and wonder if I should keep driving it - I seriously doubt that I could react appropriately if the vehicle went to full throttle w/o warning. I would for sure step on the brake, but, according to Mr. Barr's testimony, that's the wrong thing to do. What's the right thing to do? Switch off the ignition? Ram the automatic transmission lever into reverse? Given this knowledge, what's my responsibility in the event of a loss of throttle control event and the nearly inevitable accident? Morally I can't justify laying all the responsibility on Toyota but the chances of this happening to me are very, very small.
Besides the above, I'm wondering what my car is now worth and whether Toyota will step up and replace their badly-engineered software or the entire engine control module. That would be the right thing to do, but my money is on a big consumer blow-off using the NHTSA settlement as a broom to sweep it all under the floor mats.
The thing that really puzzles me is why the popular press hasn't picked this up yet - I expect to see it splashed all over the place. It shows that software can never trump celebrities or political bloviatators.
First, Toyota recalled more than 10 million vehicles for problems related to unintended acceleration in 2009 and 2010, starting with a September 2009 announcement that it was recalling 3.8 million Toyota and Lexus vehicles because of a defect that may cause floor mats to jam accelerator pedals. The company later recalled vehicles over defects involving the pedals themselves.
(Now, curiously, 2005 Camry which was the car at dispute in this Oklahoma case has NOT been recalled by Toyota yet.)
Toyota's recalls led to lawsuits claiming that defects harmed the value of Toyota vehicles or caused accidents leading to death and injury. Toyota settled suits claiming economic losses for about $1.6 billion. That was the end of Dec., 2012.
Toyota won the three unintended-acceleration claims that previously reached jury verdicts since the recalls. The defense verdicts include injury cases in New York in 2011 and in Philadelphia in June. A Los Angeles jury in October cleared Toyota of fault for the death of a 66-year-old woman.
What's important and what's different about the Oklahoma case is that this case -- among a host of lawsuits filed against Toyota concerning unintended acceleration in its vehicles -- is the first in which the plaintiff has laid the blame squarely on the electronic throttle system.
As a result, this is the first trial that any jury actually heard expert witnesses such as Michael Barr explaining the software gllitches (combined with other factors) that led to the unintended acceleration.
The experts' findings (laid out in Oklahoma case) in fact led to the one-billion dollar settlment for the economic losses, late last year. But since the case was settled (never went to a trial), the experts' report or testimony has never been made public, and no jury heard the case whose focus was on the electronic throttle system.
Because this case went to a trial in Oklahoma, now for the first time, the public had an opportunity to hear and read what were discussed during the trial. It's a matter of public record now.
The general press probably hasn't had time to look into all the details about the embedded system software malfunctioning.
But watch for the upcoming trial nex tweek in federal court in Santa Ana, Calif.
Attorneys for the plaintiffs in that case plan to argue that defective software caused Camry to accelerate and crash into the side of a Georgia schoolhouse.
Good questions, sixscrews. From the transcripts, if I understand them correctly, if the car goes into sudden uncommanded (by you) acceleration, you can brake, release the brake for a few tenths of a second, then brake again. But like you say, ramming the shift into reverse should also do the trick, an/or shutting off the engine.
As for Toyota, assuming what we all think we understand is factual, I'm not sure why they can't send out update kits to install. Some of this would be just new firmware that splits out apps better. And they would also want to reapportion tasks to different processing units, to split up this infamous Task X to different hardware (split out the monitoring and fail-safe functions). I'm not sure why this can't be done as a recall. Without any inside knowledge, it seems to me that once the new software architecture has been figured out, replicating it in cars out there now should be doable. We do this type of firmware update, remotely, one our systems, very frequently.
Redeveloping or just patching buggy code is not as simple as it sounds. I remember a New Yorker cartoon from the '40s showing a customer at an auto repair shop asking whether the mechanics put new carbon in when they took the old carbon out (this was from the time when engines normally developed heavy carbon deposits and required tear downs of the top end to remove them).
In the same way, I always told my development teams that fixing known bugs necessitated inserting new bugs - a good Monday-morning joke but not really that funny. However, bugs are inevitable and, as many commenters here have pointed out, there are many, many methods of detecting and remidating them - but they only work if the development team is willing to accept their own fallibility and has the support of managemnet to rework something that is supposed to be finished.
All repairs are fraught with risk - will the repair be done corectly? will further damage be done to the unit during the process?
I would be very leery of taking on a 'new and improved' version of the engine conrol code in a Camry covered by Mr. Barr's analysis. As many other comments have pointed out, the system at present has no independent throttle system fail safe and relies on a single CPU running multiple threads to manage a plethora of processes, some of which are trivial and others of which may be life threatening. I would rather see a full redesign of the control system, reviewed by an independent group and compliant with existing real time standards rather than a redux of the existing code labeled with a 'trust us, we are engineers' sticker.
I don't see this happening - the models in question are ageing and Toyota is in business to make money, not to make people happy or safe. IMHO they will try to sweep this under the floor mats and spend more on advertising and lawyers to cover their problems rather than investing money in strengthening their engineering teams and compensating those injured by their incompentence. That may happen (strengthening their engineering teams) but it will be a reactive response. After all, they are making lots of money selling hybrids [subsidized by US taxpayers, by the way] and other technological wonders that maze the minds of those who should know better.
I don't know what the long term solution is, but I, for one, would stand up for criminal penalties for faulty products - kill someone with a known faulty product, you got to prison. That's the case with drivers who injure construciton workers - why not for manufacturers who make products that injure and/or kill people? These are not exactly equivalent cases, but the past is littered with catastrophes that resulted from the hubris of designers.
Recently Boeing was forced to ground an entire generation of new aircraft due to a battery control problem. Why doesn't the NHTSA have the authority to take faulty cars off the road? First, the problem is not as severe - a failure at 35,000 feet is much more likely to be catastrophic than one on a public highway. Still, as control systems take over more and more of the functions that a driver assumes are 'theirs' there is a creeping loss of control. Perhaps we did not realize this or were unwilling to face up to it, either as a community of responsible engineers or as a nation that relies on a governement agency as the last defense against disaster (regardless of the attempts to blame government for our own failures whether intentional or accidental).
Think of the Titanic 101 yeara ago - advertised as unsinkable with 'watertight compartments' - except the compartments did not extend the full height of the ship so that when one filled it overflowed into the next - a progressive system failure that killed more that 1,500 people. We have many more recent examples - Maverick bumper bolts that ruptured fuel tanks, Explorer tire failures that resulted in rollovers (I don't mean to single out Ford but those came to mind easily).
Back when I was making things that went out into the field I often woke at night wondering if something I had done or compromised on would result in a disaster. Fortunately there were no personal injuries connected with anything I designed or built (perhaps I should qualify that with 'as far as I know') but there was a certain amount of luck there as well as engineering horse sense.
As we move further into the age of autonomous vehicles it is time to think about what role software plays in things we are familar with - cars seem innocous and common, but the technology inside them is changing rapidly and the things we assume about the cars we drive are often no longer true - the gas pedal does not connect to the carburetor (there is no carburetor [what's a carburetor, Daddy?]) nor does the brake pedal necessarily connect directly to the hydraulic system - these are just inputs into a processor that is built w/o standards or any oversight.
Enough - this has to stop somewhere before we have 1,500 people killed at the same time by one software bug.
Welcome to the twentyfirst century. And act like grown ups, because that's what we are, no matter what images we hide behind.
@sixscrews, sound analysis, great post. Thank you.
Recently Boeing was forced to ground an entire generation of new aircraft due to a battery control problem. Why doesn't the NHTSA have the authority to take faulty cars off the road?
A very good question.
As Michael Barr pointed out:
NHTSA needs to get Toyota to make its existing cars safe and also needs to step up on software regulation and oversight. For example, FAA and FDA both have guidelines for safety-critical software design (e.g., DO-178) within the systems they oversee. NHTSA has nothing.
That "NHTSA has nothing" comment makes me speechless.
...Perhaps we did not realize this or were unwilling to face up to it, either as a community of responsible engineers or as a nation that relies on a governement agency as the last defense against disaster.
In many ways, the public has not realized the extent of software defects Toyota introduced in the electronic throttle system. Much of the discovery by the experts' group had never been made public until the Oklahoma trial.
This "flip-bit" situation reminds me of an AT&T problem several years ago. Their long-distance phone system went down entirely. The controlling software had been running without problem for many years. Upon examination, it was determined that one line of code that had never been executed in the previous years was finally executed because all the parameters leading to its execution were met for the first time. That one line of the source code was missing a semicolon at the end of the line of code! That's all it took to bring the entire system to its knees.
The code was not reviewed ? Although it sounds funny but the implications was huge ... As a newbie in the embedded field "the bit flip that killed" tells me never to be complacent and make sure the code is peer reviewed before release
IMHO Toyota should be forced to publish the complete source code of the faulty ECU, as an object lesson to the industry. Clearly it's not suitable for commerce. I can't see how confidentiality can apply when people die. Besides, the threat of having your code exposed might be a better incentive to do better than the risk of dead customers. ... only half joking.
>Toyota should be forced to publish the complete source code of the faulty ECU
Good idea, along with flow charts and schematics. Not just Toyota, but all manufacturers whose code is so crucial to public safety. There was a time when a backyard mechanic could see exactly what was happening in the workings, not any more when so much is hidden in silicon.
And, yes, there should be a legal requirement for a big panic button kill switch that shuts down all software functions and returns complete mechanical control of the vehicle to the driver. This is the only effective "fail-safe", assuming of course that the driver is not drunk, or texting, or both.
After a close reading of the testimony I was wondering if the sections marked [REDACTED] were not marked in this way due to trade secret but rather due to the extreme embarassment of the software developers who developed this code base. [REDACTED] may perhaps be read as a fig leaf concealing worst practices(*) from the community rather than a protection of the rightful IP of a set of hard working engineers.
Perhaps all of us who own these vehicles should join togehter in a new Engineering Society, the Community of the [REDACTED] and label our vehicles, and, perhaps, other efforts, with the proud term [REDACTED].
What say you?
(*) I have always wondered why, in a community that celebrates 'Best Practices' there is not a contrary group that celebrates 'Worst Practices.'
NASA's Orion Flight Software Production Systems Manager Darrel G. Raines joins Planet Analog Editor Steve Taranovich and Embedded.com Editor Max Maxfield to talk about embedded flight software used in Orion Spacecraft, part of NASA's Mars mission. Live radio show and live chat. Get your questions ready.
Brought to you by