Toyota recalls: Deeper engineering implications

2/9/2010 6:27 PM EST

Excellent points. However, I think you are underestimating the software reliability problem. I believe that software is almost always to blame, even in situations like the Toyota sticky brake pedal where the hardware is defective. If the software was properly written, the problem would have been discovered much earlier and accidents would have been prevented.
*
The problem with current safety-critical software systems is that manufacturers are reluctant to deploy sophisticated software because they know that reliability is inversely proportional with complexity. And the the reason for that is that the basic software paradigm used by the industry has not changed since the days of Charles Babbage and Lady Ada.
*
The Turing Computing Model is inadequate for modern applications because 1) it does not address parallelism and 2) timing is not an inherent part of the model. What is needed is a new model that makes it easy to determine whether any two operations in a program are simultaneous or sequential. In other words, software needs to be more like logic circuits and less like algorithms. I've been calling for a deterministic, non-algorithmic, synchronous, reactive software model for ages. Nobody is really listening because computer science is still controlled by the same baby boomer generation who got us into this mess. It's time for a change. Go to the link below if you're interested in the future of safety-critical software.
*
Why Software Is Bad and What We Can Do to Fix It:
http://www.rebelscience.org/Cosas/Reliability.htm

2/9/2010 6:32 PM EST

Also, the parallel programming crisis is a direct result of our infatuation with the Turing Machine and multithreading. We must switch to a new software model that is inherently parallel and does not use threads. Threads are evil.
*
How to Solve the Parallel Programming Crisis:
http://rebelscience.blogspot.com/2008/07/how-to-solve-parallel-programming.html

2/9/2010 8:52 PM EST

I've been designing and developing high reliability systems for almost 30 years and I have to agree with Mapou for the most part. Systems in these sort of real-time systems have to be designed with parallelism as part of the model, not as an afterthought. They also have to be designed for failure. All systems will fail. Its what they do when that happens that is important. Most of these automotive systems are designed to not fail, which is contrary to reality. As one really bright economist once said (quote not necessarily accurate), "Normally rational markets can behave irrationally longer than any investor can stay solvent". That sentiment applies to these systems as well. Any model that does not accommodate "irrational behavior" of the system is doomed to catastrophic failure!

2/9/2010 8:56 PM EST

FWIW, that comment about "rational markets" was applied to the failure of Long Term Capital Management a few years ago, sort of the harbinger of the mortage and capital market meltdown of the past couple of years.

2/9/2010 10:41 PM EST

I am sorry, but the failure is not that complex and within the realm of expected. Stuck throttle was a common problem in auto racing and back in the bad old days before brakes were well engineered it could end quite badly. The failure appears to be a problem in the throttle by wire or cruise control which applies full throttle. Consumerreports and others have already done the test of applying full throttle and brakes at the same time. Some but not all other car makers with fly by wire have included a throttle cut under braking, either as software or separate hardware. In most of the cases the brakes were barely able to overcome the engine if the car was at speed. So the brakes are too small and the modern throttle did cope with an old problem. And don't even start on the fact that the keyless push button start needs a big red kill switch as covered by the yestertech in NASCAR. Easy to do, easy to predict. Generally Toyota appears to build well solid if boring cars, but this is a Ford class failure, e.g. Pinto gas tank, Explorer exploding tires. I find it hard to believe that someone at Toyota had not noticed the problem prior to the recall and they are guilty unless they can prove their innocence.

2/10/2010 11:53 AM EST

The brake should always override the throttle for reasons too obvious to state.

For brake failures a secondary system could be the hand brake taking into account hand brake pressure and wheel speed. When stopped the handbrake will function as usual, however above a set speed, e.g., 3mph, the main brakes will operate.

Finally the last resort is cutting the engine electrics & fuel supply which works well on stick shift cars. It can seem daunting at first try since normally the ignition key is only turned when the car is stopped, however the car continues to behave as expected, the main difference being that one cannot now of course accelerate. Power steering continues to function well until speed has dropped to about 4 mph & even then steering is still possible. A stick shift car can be brought complete halt down a relatively steep hill using the gears alone.

Practice is important, and any practice is better than no practice.

2/10/2010 2:56 PM EST

The hand brake is not strong enough.

Just cut off the fuel supply. Most cars have an electric fuel pump. A simple switch can halt the car.

2/10/2010 3:36 PM EST

What bothered me so much about the developments in this crisis was the complete refusal, initially, of Toyota to even allow for the possibility of software/firmware/electronics being involved. If it turns out to be nothing more than the sticky potentiometer-pedal and before that the interference from floor mats, the idea of not even allowing for the possibility of anything else strikes me as dysfunctional at best.

Mapou's general direction is astute it seems to me. I hope that it gets more press and real action, whether or not fallacies of software testing wind up being involved in these specific crises or not.

Brad Wood

2/10/2010 4:07 PM EST

I'm sure there are plenty of fingers to point:

1. Time-to-market pressure: There simply isn't enough time to test pre-production units through all possible scenarios. Despite careful planning, technical programs almost always experience schdule overruns, and at some point, things aren't allowed to slip any further. You go with what you have.

2. Cost pressure: Technical programs are often understaffed, and underskilled with less-experienced (ie, cheaper) engineers.

3. Complexity: It's difficult/impossible for the human brain to comprehend/predict all possible scenarios, sources of failure, and appropriate fail-safe actions. These get even more complex with multiple-simultaneous failures.

4. Material costs: Subcontracted manufacturing to the lowest-cost provider may cause problems later on. At some point, corners get cut to reduce costs, and those cuts can affect quality & reliability.

5. Competitor-bashing: Are Ford, GM, Chrysler, etc offering any assistance to Toyota to resolve this problem ? Hah! I doubt it; Could they be stoking the bad PR fire (behind the scenes, of course) against Toyota ?

2/10/2010 4:18 PM EST

green_ee wrote,
*
"Complexity: It's difficult/impossible for the human brain to comprehend/predict all possible scenarios, sources of failure, and appropriate fail-safe actions. These get even more complex with multiple-simultaneous failures."
*
I agree with you about the impossibility of the human brain to predict all possible consequences. However, I am convinced that, given the right software model and adequate path discovery tools, all possibilities could be discovered. The reason that this is not done in practice is that our current software model is hopelessly flawed. There is a correct way to do things but it will require courage more than anything else to switch paradigms.

2/10/2010 11:08 PM EST

Software issues are rampant. As we begin to depend on SW vs HW to stop a car safely we buy into the same BS we see at Microsoft. Lousy code and pitiful support.

Get real folks. Algorithmic experts are the greatest. But given my experience with them in computational litho world === I gotta wonder if my car will stop!

jimmymac

2/11/2010 12:51 AM EST

The issue is every sigma adds cost and complicates the entire design to delivery team's life. We need to find a simpler quality system which leave engineers to do more technical work than too much of documentation work. I think Toyota need to do a root cause analysis and pin point exactly what part of quality system lost control and why?, - a true 'why' analysis is need, after all having so much of quality system implemented.

2/14/2010 5:41 AM EST

There is no doubt Toyota's engineering and test capabilities are very competent. Their manufacturing processes and quality control are probably second to none. However, in all the talks about root-causes, more testing, more robust designs, etc, etc, there is little mention of failure mode effects analysis (FMEA).

If the failure modes are identified and mitigations implemented for the most severe ones, the probability of a vehicle with a stuck gas pedal killing all its occupants would be much less than what we have experienced.

This is a single-fault-condition that could have been identified and mitigation(s) put in place to prevent (or at least significantly minimize the probability of) an accident. It is unreasonable to use zero-defect as the first line of defence in a safety related situation.

Yes, this will add engineering time and cost to the development budget. But, I am fairly sure this cost is far less than what Toyota is spending on the recalls and PR damage control.

2/16/2010 1:37 PM EST

Bill, you're right about the law of large numbers, and you're right about the pundits not knowing what they're talking about. My only quibble would be that for a mechanically sticking accelerator pedal, this particular company had a gigantic experience base with getting that right in the past. What did they change such that it's not right any more? Did they have a good reason to mess with something that was working? If so, I agree with you. If not, then they took a risk which now seems not to have paid off. Was it still a risk worth taking? Only Toyota could know that. This is what the pundits are missing -- engineering is the art of combining what is known to work with new elements so as to make something that is better than what existed before. If you take risks, as engineers must, then sometimes you'll lose. The only question is whether it was an intelligent gamble or a gaffe.

2/16/2010 1:50 PM EST

Just one little nagging thought: what if the 'sticking throttle' problem is NOT due to what the a 'sticky' mechanical. Or is not totally due to that problem source. What if this is a multiple source problem (e.g. the carpets and the 'sticky' pivot point and ??????). There is one fix that should, in my opinion, be part of the solution - a software patch to link the brake pedal to the throttle mechanism. Some other cars use this - why not Toyota? They are patching problems that might cause the throttle to stick - they do not seem to be tackling the question - what do you do WHEN the throttle is applied AND you are trying to brake! This should be a component of any fix for this problem.

2/16/2010 10:47 PM EST

As you may know toyota has two major recalls, one for sticky gas pedal(mechanical Failure) and the other regarding the interference of the floormat with accelerator pedal.The last stage in floormat recall is to reflash the ECM. The new software will cut off the fuel from engine if the both pedal pressed simultanously. This feature is already used in Camry hybrid software so the reflashing would not be necessary.

2/17/2010 9:33 AM EST

I respectfully disagree with your conclusions. Based on Toyota's analysis of the accelerator pedal issue (and the fix), this is attributed to a materials wear problem. The mechanical friction fixture that is causing the "sticking" exists solely to provide pedal feedback.

I don't intend to offend, but as a practicing Engineer, I have an obligation to the public, especially where safety is concerned. Dismissing what appears to be, in fact, improper testing by trying to make a "law of large numbers" argument just discourages folks from taking home a valuable lesson. The tuition has, unfortunately been paid with human lives. This isn't a statistical outlier... This is a materials wear issue. Toyota made a mistake. Plain and simple.

2/17/2010 10:32 AM EST

I disagree aiki10. The material wear out problem should be more prominent on older car which is not the case here. It is more toward the electronic control system. You could not catch up 100% possible glitch problem in the production test. You could lower it down to few DPPM but when production number is in the millions, that is quite a few cases and exactly is what we see in Toyota.

2/18/2010 9:26 AM EST

To assume that Toyotas accerator problem is a mechanical problem is naive. Software bugs can cause similar issues, but your comments on performing enough testing to identify every possible problem is valid.

2/18/2010 11:26 AM EST

It is a credit to the Toyota management that the timing of the announcement, solution and recall was so well executed. I heard discouraging comments like "its been over a week, and Toyota is just now supplying dealers with correct parts". Now I don't know about you, but my reality is that if a problem surfaces and 8 MILLION "fixed" parts are delivered in a week, that's nothing short of a miracle.

Toyota was exceptional in containing the news, managing the release, and suffering the consequences. No one blabbed, no one created panic (until that Congressman got in front of the camera), and a "solution" was delivered in seemingly record time.

2/24/2010 3:45 AM EST

Can I offer the following excellent article written by Keith Armstrong:

http://www.nutwooduk.co.uk/downloads/Toyota.doc

Best regards

Tim Probert
Principal Engineer, Renishaw plc

2/24/2010 10:53 AM EST

Keith may be an expert, but examples he uses are poor. His analogy for proving whether a lamp has been on? I can think of several methods, the most obvious that I use is, touch it and see if it's warm.
I guess I am also wondering why if Officer Saylor was so experienced, why not just switch off the ignition?
My daughter asked me about this recently, and I went through what I thought one should do, stomp on the brakes, shift to neutral, switch off engine.
I must admit I felt that holding the brake down would slow and stop a car, but maybe I'll have to try it myself.
I certainly don't think cars should be unreliable in an unsafe way, but after reading Keith's paper, I could be paranoid about getting in a car.

2/25/2010 8:46 AM EST

For the millions of cars on the road, its really amazing to me that there are so few serious problems like this. The last line about the law of large numbers being tough to work around is so true! Well said.

3/3/2010 3:10 PM EST

We are not really certain that the Toyota sudden acceleration problem is a sticking gas pedal. That theory is a quick and easy one, not that hard to fix and not that indicative of a faulty engineering process. Check the demo by the two NASA guys on yuo-tube, where they duplicate the acceleration with a momentary short circuit of two of the throttle control wires, and no fault code is generated. This proves that there is at the very least, a defect that does not record a fault that causes sudden acceleration. Consider the implications of THAT for a while. The reality is that they may well be fixing something that is not the problem.

3/3/2010 6:36 PM EST

As others have noted Toyota has been running around like achicken with its head cut off trying to blame this on anything but the software.
Is the code being changed when these cars go in for recall???
Someone should determine the code version and total size before and after recall to see if they are doing any changes.

This problem is really about good engineers being able to speak there minds and be encouraged to critique others designs.
Now everyone just keeps their mouths shut bacause everone has seen the outspoken engineer thrown out on the street.
Group think has taken over everthing now and if one wishes to keep feeding their fgamily they keep their mouths shut so the direct deposit continuos to flow.
Bill are that nieve? have you not seen all the changing excusses coming out of toyota?
Not to leave the conversation without adding some hope and a potential solution. why not have the code return to zero level each cycle for a small duty cycle period and only return to set point if no error detected like brake applied?
If those drifters complain they can add a momentary switch override for this brake error with large throttle etting condition.
Also a large changing (quick = panic) braking force could be used to zero out the throttle set point as an override. This would allow for suttle braking with some throttle applied that may be needed for certain driving techniques.
This could have many levels of override to prevent this in multiple wqays as it should be.
The redundant pot mention is a must also.
whach out for groupthink it bites sooner or later and this has resulted in deaths.
who wants to be next?
Any volenteers?

5/15/2010 5:29 PM EDT

Redundancy is a great tool to avoid failures, but several points must be noted in response to earlier posts:

1. Redundancy is only useful if the errors of the redundant systems are not correlated. Hence, two identical sensors is often more reliable than one, but that may not be the best solution if both sensors fail simultaneously for the same reason. Really good failsafes have uncorrelated failure modes.

2. A modern automobile is already pretty expensive without redundant systems. The consumers have long decided that they would prefer to have creature comforts first and safety only if it is really inexpensive or mandated by law. I bet that every one of the vehicles that failed to stop had a CD player and probably even a GPS. Nobody opts-in on the driver-side airbag. If it is not mandated as a safety requirement, it won't exist on the majority of vehicles. The same is true for all safety devices.

3. Drive-by-wire may be spooky to some, but it is not the problem. Don't throw out the baby with the bath water. Drive-by-wire is an excellent way to improve the functioning of our automobiles. It is the lack of proper failsafes that is the problem.

4. Turning off the ignition switch (or having an equivalent function when the brake pedal is pushed really hard for a really long time) would only kill power to the electric motor. Breaking the connection between battery and the motor would not necessarily brake the vehicle (unless a resistive load is simultaneously switched across the motor windings). Both mechanical and electrical failsafes could be implemented that act when the drive-by-wire system fails. That is where Toyota failed to protect the consumers of their vehicles.