Breaking News
Comments
Bert22306
User Rank
CEO
Like Toyota?
Bert22306   3/4/2014 1:53:26 PM
NO RATINGS
I was hoping EE Times would publish an article about this, but quite honestly, I thought it would be Junko!

Could this be the same sort of phenomenon as the Toyota unintended acceleration story? The numbers are so small, compared with vehicles of that type on the road, that the issue only seems like glaring oversight when looking back? Doesn't look good for GM, that's for sure.

At least this time, it didn't take a high visibility lawsuit!

junko.yoshida
User Rank
Blogger
Re: Like Toyota?
junko.yoshida   3/4/2014 2:51:52 PM
NO RATINGS
@Bert, I spotted yesterday that my colleague Chuck Murray over at our sister publication Design News did a great job reporting on this -- so I decided to move this story over here at EE Times.

The blame, i think, actually goes to not just GM but also to NHTSA for the lack of their oversight. 

But more relevant to our community and even more fascinating about this story is, as Chuck wrote in his story, this could be a good case study for how big manufacturers [like GM] handle the smallest details of design [or they didn't actually handle it very well]. 

 

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 4:51:22 PM
NO RATINGS
Not like Toyota, the unintended acceleration was not a fault of design in electrical/mechanical/software and was almost entirely the fault of the drivers.


Crashing or having keys on your keychain or even just bumping the keychain as in GM's case is not something you could exactly say is entirely the fault of the drivers. (Crashing is arugable, but the heavy keychain's are not as what kind of car requires you to have just the car key by itself) They also had a design revision of the out of spec switch so they clearly knew it wasn't performing properly even as designed.

 

GM has a design problem that should have been easy to identify and fix but something went wrong and it wasn't done in a timely fashion. Also shouldn't the airbag control not disable the airbag's if the car is still moving (Software issue is also unlike pedal override which has handling tradeoffs in that you can't use both on a steep hill situation but most drivers don't know how to do that anyways so whatever)? What happens if you crash in an unintended acceleration case (pedal misapplication applies to all brands) and the driver turns the car off intentionally but still crashes. (Pedal override would also work but so would neutral)

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 5:24:32 PM
NO RATINGS
Not like Toyota, the unintended acceleration was not a fault of design in electrical/mechanical/software and was almost entirely the fault of the drivers.

Not so, right? After extensive investigation, unintended acceleration was found to be possible in Toyotas if a particular supervisory engine control app crashes. Depending what is going on when this app crashes, it was found that such a crash could lead to a wide open throttle command.

A court case in Oklahoma, extensively covered by Junko Yoshida, brought this to light.

http://www.eetimes.com/document.asp?doc_id=1320013

That too was a long time before being figured out.

The similarity in both cases is that while the drivers did have some measure of control, it is very different from what a driver normally expects. For example, if the engine is off, or at wide open throttle, you no longer have vacuum assist in the brakes. So you really have to push on that brake pedal to get the car to stop. This happens whether the key has gone to the acc position (wheel unlocked but engine off), or whether the throttle is stuck wide open. (Yes, the brakes do overpower the engine.)

An additional twist in the Toyota case was that when this supervisory app crashes under certain unfortunate conditions, in order for the brakes to work at all, the pedal had to be pushed, released for a few tenths of a second, and then pushed again, to get the brakes to work.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 7:05:02 PM
NO RATINGS
The DOT testing showed even under a single suprivsory control failure the engine control has at least two independent paths of control which would prevent an unitended acceleration event even under the case of an ECU failure.


Study of the event data recorders (Independent of the ECU) found that in all documented cases except ones involving entrapped floor mats it was the accelerator applied and not the break pedal. This data is recorded from dual redudant sensors which in some models could have failed in a double failure condition but this wasn't found either to be the case on inspection after the fact. Quite simply it was the drivers for the most part and this has happened in the past with other mfg as well.

http://www.dot.gov/briefing-room/us-department-transportation-releases-results-nhtsa-nasa-study-unintended-acceleration

Also in the report from DoT in partnership with NASA they bombarded the ECU with intense interference to attempt to induce a failure as you mentioned and all failure modes resulted in limp mode or stalling.

The NASA report had full access to hardware and source code and used in the cars in question. Also the use of the terminology of an control app isn't really correct as the ECU isn't like a typical operating system and basically runs one "app". Even in the even of a open throttle situation another failsafe exists in the software to control engine speed by fuel restriction. The pedal has double position sensors and pedal release sensor act as the redudant/diverse sensor system. It would arguably be more fail safe than a throttle cable strapped directly to the engine throttle as there are some cases where the wire you get jammed mechanically without electronic failsafes.

The only court case to succeed in any respect was the class action not about any failures but a class action due to the loss of resale value due to perceptions not facts.

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 7:12:09 PM
NO RATINGS
Sorry, Adornao, it appears your references are out of date. You might want to read these two articles to see what I'm talking about:

http://www.eetimes.com/author.asp?section_id=36&doc_id=1319910

And then the final outcome:

http://www.nytimes.com/2013/12/14/business/toyota-seeks-settlement-for-lawsuits.html?_r=0

This occurred between October and December of 2013.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 7:18:58 PM
NO RATINGS
Report wasn't released to the public so I don't/can't consider it based on second hand reporting. In addition the software/hardware model showed how illeterate the report was as Task X the big thing that failed isn't like an app it is probably the main CPU's main loop. Even if the entire main CPU failed the secondary monitor CPU can detect independently the throttle commands to the sensor inputs and put the car into limp mode unless they are suggesting a SEU could simultaniously occur in both independent CPUs which is getting a bit improbable.

This case was also later settled out of court so it wasn't "successful" as the outcome in not avaible for us to see it was unknown and the only case to proceed is the class actions which have proceeded without secret settlements.

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 7:28:31 PM
NO RATINGS
You aren't even reading the excerpts reported here. Once again, EE Times covered the case extensively, even if the general press did not. Exerpts from the court hearings were published here and I'm sure you can still go and read them. It was demonstrated how Task X death can cause the throttle command to go full open. If this was not possible, it could not have been demonstrated.

In fact, according to this:

http://www.autoblog.com/2013/12/16/toyota-unintended-acceleration-cases-halted-judge-settlements/

it ain't over yet.

I see this and the GM case being very similar, in their effects on the driver experience, should the problem occur.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 7:38:02 PM
NO RATINGS
Did you even read that news article before posting it, unlike the undiscolsed settlements which we can't judge as being won or not the "economic loss cases" where publicly settled for 1.6Billion (Which is probably a win for the class action).

Until we know what thoses select cases settlements are the only one's to proceed to a winning state are the one's we know about it is pure conjecture otherwise.

Task X can command full throttle it doesn't matter as the monitor CPU wouldn't agree and would shut Task X up (And Task X is probably the main CPU entirely as it's sole task if you read the public NASA report is to control the throttle position, while the monitor CPU has the A/D converter (Not a task) and additional fail safe sensing (Not on the main CPU) on top of the main CPU failsafe code to consider the event that the main CPU is corrupted and it's failsafe's are disabled.)

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 7:41:32 PM
NO RATINGS
Did you even read that news article before posting it,

I followed the series of articles on EE Times very closely, yes, in those months in 2013. There were many exerpts from the Oklahoma hearings posted.

Did you read them yet? This is not pure conjecture anymore. Do yourself a favor and read the transcripts.

The NASA report is way out of date.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 7:48:30 PM
NO RATINGS
No report, no data, no evidence, no settlement numbers, no nothing. I base my information off what I can see and evaluate with my own understanding not through people who are invested in a case and in a fight for millions of dollars. When everything is under wraps the old way out of date but public reports will have to do.


As I have said repeatedly and you seem to ignore main CPU task X can do whatever it wants as the hardware as a monitor CPU to watch task X so unless they also have a quote for that which you haven't provided with technical info on how task X tricks the monitor CPU even though it is just reading raw sensor data and seeing if it makes sense (100% dual throttle sensor with 0% dual pedal sensor doesn't make sense and no amount of Task X bubling on the main CPU would induce a false ok condition on the monitor CPU)

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 7:14:19 PM
NO RATINGS
The case mentioned in your article was settled out of court and no useable information was released but a SEU and EMI/RFI would not have caused a uncontrolled acceleration as the electronic throttle is fail safe in that there are more than one controller monitoring the system and multiple redudant sensors. The case also had them claiming that a surging car could not be stopped which is also false.

Task X is probably the entire main control loop (We don't have that report and I doubt it was peer reviewed or anything) of the main CPU which isn't the fail safe in reality as most of the main protections are in the main control loop but should it fail to a SEU or external interference a secondary monitor CPU can detect a main CPU fault and drop the car into limp mode.


Unless they have the event data recorder logs to prove it everything in that case is conjecture and all the evidence upto now has shown it was the drivers at fault for the most part. History has proven this with indpendent manufacturers with decades apart and totally different software/hardware/electronics/engineers with the same pattern in these cases.

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 7:17:49 PM
NO RATINGS
You owe it to yourself to read the evidence. The throttle is apparently not fail safe, when Task X dies. That was a key finding. And the lawsuits were not for lost resale value at all, in these 2013 cases.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 7:25:53 PM
NO RATINGS
Without an EDR (Event data recorder) report their claims are baseless given the mass of publicly released evendience and statistics to the contrary. Arguing that the main CPU might fail is not relevent when independent safety control exists. Sure the monitor CPU can't run the car properly but it can detect that very situation they are claiming which is something the DOT/NASA study proved works. (Old doesn't mean out of date)

 

Unless your claiming that two diverse CPUs fail at the same time I find it hard to believe you think an unreleased report is credible. Consider the hardware design before believeing in second hand information a main CPU failure can't take out the secondary CPU at the same time. And if you do think that then all safety related systems which rely on two processors are now considered unsafe? Technically their non-lockstep model dual safety system is safer than a shared lock step system in that the two processors must be running different code so even if "Task X" fails "Task X" can't exist on the other limited CPU.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 7:32:28 PM
NO RATINGS
And finally back to the article GM knew about the problem and made a fix but then decided not to implment it even after the same statistical and warentee information which showed serious reliablity issues to the point where they even wrote a notice to dealers about the switch but failed to recall it.

The same warentee and statistical evidence include strange EDR reports of cars switched off or in accessory mode during a crash which are a stark difference between Toyota where all the EDR reports point the finger at the driver (EDR is not part of the ECU main CPU task X by the way and is usually in a seperate module with sensor data being fed to it and is typically part of the airbag controller which before fully fledge EDR has a simplified version of an EDR)

Thats how different it is between GM and Toyota. One has statistics, their own engineers, warentee information, 13 dead, countless crashes, dealer warning, a recall on all affected parts, ...

Toyota hasn't recalled or modified the ECU software short of adding the pedal override which isn't ideal if you are a classic driver as you can't press both pedals at once. (Although most won't understand why/when you would do that)

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 7:37:04 PM
NO RATINGS
By the speed of your replies, you seem to be on constant transmit mode.

The Toyota case is not over. Read that autoblog piece I posted. The Toyota case also has "wrongful deaths" statistics, but the mechanism for the failure mode was not discovered until last year, fourth quarter.

Once the mechanism is discovered and demonstrated, it makes no sense to deny its existence.

Your lighning fast replies tell me you are not reading what is available now.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 7:41:07 PM
NO RATINGS
Give me the full report that proves it and then I will consider it but if I don't know the technical details as an engineer I don't think it holds much crediblity.

I doubt the hardware design could mutate between the reports so if what they claim task X on the main CPU can cause the failsafe throttle control to fail my rebut is that the design has a monitor CPU that's very name sake is to monitor task X on independent hardware.

You also failed to read the very article you posted and the only number comes from economic loss cases which is exactly what I'm referring to.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 7:44:44 PM
NO RATINGS
Again unlike what you asked GM is clearly at fault and at all levels knew about it the most we can say is maybe some strange intense solar radiation knocked out both main (task X) and monitor CPUs without causing them to fail to run the motor while ignoring sensor data at the same time. I think if that is the case then I feel safe driving that car as the probablity of that happening hasn't be demonstrated in many many years unless it is a triple SEU whereby the independent EDR also gets corrupted but legitamate looking sensor data. Or maybe all the sensors read full throttle because they all simultaniously move smoothly to valid limit positions for some foot application reason which has been proven to be the main bulk of the incidents across mfgs. (Note how I said bulk and most as I'm sure one day even a quad redudant system could fail regardless of design but that is highly improbable)

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 7:48:50 PM
NO RATINGS
The software expert showed that there was too much riding on Task X, and that when it crashed, it didn't always reboot itself. Once again, the transcripts that explain this ARE available in the EE Times series of articles, and I'm not sure why you aren't reading them.

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 8:02:38 PM
NO RATINGS
Here are two articles you MUST read, before making hasty pronouncements:

http://www.eetimes.com/document.asp?doc_id=1319966

http://www.eetimes.com/document.asp?doc_id=1319936

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 8:26:47 PM
NO RATINGS
Also if you can't read the little side bar on the side to tell that I'm not a EE|times reader you seem to be mistaken that I would have read this site's coverage.

 

I guess you care more about transcripts than engineering so its a bit strange. No actual evidence was presented in those very transcripts you provided. Unlike the GM case which has deaths with strange EDR sensor values the Toyota cases have normal EDR reports with strange user values of the dual redudant pedal sensors), even if task X fails and subCPU fails as the sensor values would either be contridictory or non-sensical but nothing of that nature has surfaced and they are just grasping at millions of dollars if they can't prove that it actually happens in practice. I don't want to go blaming drivers for their own deaths/injuries but it is unfortunate that people couldn't just admit they pressed the wrong pedal but human factors is much harder/variable than electrical/mechanical design.

 

I'm open to accept real hard non-redacted proof but a debugger causing a failure isn't a real failure and not a single EDR I know of showed it happened in an accident which in GM's case many showed strange contridictory sensor information.

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 8:34:37 PM
NO RATINGS
Didn't Task X death also affect EDR recordings? I think that came out as well. I think the worst aspect was that applying the brakes would not always work, unless they were pumped once. To me, that's the worst effect. And after that, the lack of power assist at full throttle, of course.

In the GM case, where the ignition went off, the brakes would instead continue to work, but without power assist. The biggest issue with the GM case is that the air bags wouldn't deploy when the ignition is off.

So in my view, both companies have done the same thing, including waiting a decade or so before addressing their problems. Although it's POSSIBLE that Toyota really didn't know they had a problem (hard to believe, when you read the transcripts, but I suppose the engineers in that program may have gone on to other programs and not obsessed over past work).

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 9:06:25 PM
NO RATINGS
Task X is part of the main CPU which doesn't have the A/D converter in it and the A/D task isn't part of the main boat load process as they said async tasks occured outside of it (interrupts anyone which likely service data coming from the monitor CPU containing the digitized sensor information)


The EDR is seperate from the ECU but would take the A/D readings from the monitor CPU and likely just sits there watching the digital sensor lines for the data between the monitor and sub CPU. It records both the pedal and throttle value and only the monitor CPU reads those and both check the values for sanity. The EDR doesn't care and would just write down whatever is going on. If the Task X is the A/D converter then I don't see how an A/D converter can be in the main CPU as it isn't in the publicly released diagrams.

The brake pump once thing is also patently false based on a number of public tests showing the opposite result with constant braking stopping a car even with full open throttle. (Yes lightly pressing the brake won't stop a car but that isn't exactly a revelation)

Also ignoring any unproven conjecture based "expert evidence" using a debugger to cause a failure just using the slamming on both pedals would stop a Camery faster than a Ford Taurus could from 70 MPH even with a depleted reserve you could still brake and slow down to a safe speed and stop if you know what your doing.

GM knew about the problem and had a fix but didn't do anything. At most assuming your right which I don't believe you are they didn't know of a problem and haven't needed to fix it ten year later so I don't know how they are similar.

Toyota never recalled the ECU software and Task X is still going strong and UA events are down to expected background levels and are almost always "pedal misapplication". So unless Toyota is killing people who try to talk I don't think there is an issue with the software. Maybe it isn't well designed and could do with some cleanup but it isn't shown to be defective but I can't prove that to 100% and neither can NASA but we can say within a wide margin that it isn't a likely cause by a long shot.

 

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 9:17:18 PM
NO RATINGS
I see no way around the apparent fact that Toyota's Task X can crash, fail to reboot, and cause the throttle signal to show full open.

I agree that under such circumstances, same as in the GM case, a totally alert driver *should* be able to stop the car. In both cases, the steering remains functional, and the brakes too, sort of. The problem is that when you suddenly present a driver with a situation he's never experienced, very frequently the driver won't have the correct and quick reflexes necessary.

The important point being, the driver needs to do the SAME THING, in the Toyota and the GM control system faults. Push really hard on those brakes. (Assuming you're right that the Toyota case doesn't totally incapacitate the brakes unless pumped once, as was demonstrated.)

GM's excuse was that they had already changed the part, although without a change in the P/N, so they thought the problem was solved but couldn't properly track it.

I would be amazed if Toyota doesn't update this Task X to absolutely avoid the possibility that its crashing would not affect the throttle in any possible way.

I've no reason to give either company a break.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 9:36:13 PM
NO RATINGS
Task X can fail if you make it fail but that doesn't really mean anything nor do they even say what bit they are flipping is it a debug value or something what is it we don't know and can't really judge how fragile task X is as it is like saying a debugger can pause/breakpoint the CPU (That better not be the bit they were flipping). Sure maybe radiation drops the CPU into debug mode and simulates a fake breakpoint and kills task X while keeping the watchdog timer happy. Conjecture yes, proof none, same as redacted testimony)

Rebooting a ECU while the engine is on is dangerous and it isn't just a PC computer running an engine. Shutting of the entire ECU would cut off sensor data to other independent controllers and could cause airbags to fail to deploy or a host of other issues with systems which expect the ECU to be running when they are up. Which would be more dangerous than 100% throttle which doesn't even happen in practice. Want to reboot a FADEC on a jet engine (Don't do that, don't even try plugging a debugger into a running FADEC on a real jet engine) Sure those are far more reliable and robust but rebooting isn't a good solution. Memory scan/encryption/error correcting codes/... (They did have memory error correction I think but a debugger would be a valid memory change) might do better. Obviously having N+1 redudancy would be safer but that would have added costs and how much additional redudancy and diverse software do you want and what does that cost.

In GM's case it is the exact opposite even functionally the car stops running when it should be running and the user doesn't want to stop or power down the car and in Toyota's case it is the driver pushing the wrong pedal.

GM doesn't have a control system fault they have a mechanical switch design fail that they knew about and didn't correct for 10 years.

Toyota has no fault and paid 1.6 Billion in "economic loss" suits and undisclosed amounts to other cases.

GM recalled 1+ million cars to replace said key part after the cars went out of production.

Toyota hasn't recalled the ECU in question.

GM the user should turn the car back on.

Toyota the user should stop pressing the accelerator pedal, or turn the car off, or press the brakes, or shift to neutral, or... (GM has one problem the switch, Toyota has a human factors issue)

GM didn't change the part their supplier did and they never recalled the switches even though people where dying in crashes where it was directly attributed due to the EDR reports.

Toyota has no part change not even a silent one, no recall, no deaths attributed to the ECU, no EDR reports, nothing just a lot of "pedal misapplication" and mechanical problems.


GM has people running a big overly complex task X which failed silently for ten years.

Toyota has task X still running its giant blob of code today without any issue. Blob code doesn't mean it is going to kill you. Might not be pretty or maintainable but it seems to work and no amount of analysis can prove/disprove a system is bug free or not.



GM has a single point of failure in a mechanical part.

Toyota has no recorded point of failure in the ECU and many mitigations even if it did occur.

 

There is no way to make an ECU bug free but that doesn't mean it will crash you car if task X stops. A properly designed ECU (Which toyota's does appear to be by history and use) can still function safely even without a kitchen sink task X.

I don't give them a break but I don't expect them to write bug free code (impossible) nor have excessive amount of redudancy in one aspect of the car (ECU).

GM on the other hand failed in so many areas in managment, engineering, supply chain control, recall, ...

 

 

 

 

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/9/2014 6:31:07 PM
NO RATINGS
Rebooting a ECU while the engine is on is dangerous and it isn't just a PC computer running an engine.

I strongly disagree. A safety critical ECU that crashes should definitely be rebooted. That's why one uses watchdog timers in such circumstances. Of course, the reboot time has to be specified appropriately, and it typically is.

If any self-respecting engineer discovers a problem such as this one, even if the event had never been known to happen in the real world, the right thing to do is to patch the software sooner rather than later. This is actually standard practice in control systems.

My bet is that Toyota either has or soon will do so. Toyota won't want to advertize that it has fixed the problem, because that would be tantamount to admitting a problem was there at all, but they could include the patch among other updates to the ECU software.

GM doesn't have a control system fault they have a mechanical switch design fail that they knew about and didn't correct for 10 years

As the article explained, GM had replaced the part years ago, and thought the problem was solved. But once again, I cannot understand bending over backwards to make excuses for one company and not the other. In both cases, issues had been occurring for 10 or more years. In both cases, the absolute number of problems was tiny, compared with their entire fleets of cars. Statistically, it was shown that the Toyota unintended acceleration problem was far worse than occurrences from other companies. So I'll accept that perhaps Toyota was concentrating too much on the carpeting explanation, and not looking more closely into software problems.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/10/2014 5:15:19 AM
NO RATINGS
If a WDT is triggered yes restart. But no restarting if some protections are lost could result in all protections being lost or even unpredictable conditions to occur. In a single channel system such as the Toyota car's restarting at the first sign of an OS fault or other problem that doesn't stop all protections is better than rebooting into an unknown condition and maybe fixing the problem. There is no secondary/revisionary channel to take up the task if a CPU would run into an error and sometimes ignoring errors is normal. In medical industry errors and internal faults are not to cause a device to reset unless it catastrophically fails to operate if there is some remaining functions then it should be upto the user to reset the device or continue using it in a degraded state. Think about a heart lung machine rebooting while it is running and it shuts off everything during a reboot cycle and loses all the parameters and starts up blank. Better try to deal with the fault as best as possible and keep as many fail safes that may still be working than kill them all and risk having nothing after. It is better to limp to the side of the road than to potentially stall the engine or cause other unpredictable effects should the restart not succeed.

The problem with your suggestion is some "expert" who I don't really think carries all that much weight with a group named after himself with him being the expert witness and some questionable programming advice sprinkled everwhere online. His books also are topical at best and as some reviews state just wrong in general or not applicable with dev kits highly overpriced or not even manufactured anymore. (One of the best review quotes for the embedded systems dictionary is that it is suitable for managers..., not that google can't do that for you) A self respecting engineer would look at this and say there is no evidence of a problem with the ECU and if it really is so hard to maintain why fix something that ain't broken (They might end up causing more bugs and actually break something). Sure maybe it doesn't fit his coding standards but juding by what he posts online he doesn't fit the coding standards of others as well. He views optimization tricks and workarounds that are useful in various situations and horrible advice and never should be used similar to saying Goto is blanket evil when what is assembly have a lot of.


Toyota hasn't recalled the ECUs and there isn't any statistics I know of that actually prove that there is a problem the media inflated self reported incidents are not a hard statistic. (The EDR reviews are)

GM did replace the part years ago they tried to hid it replacing it would involve a recall. They didn't do this till now and using a dealership alert isn't what I call replacing a part it is called covering up a big management and engineering fail on a scale so large it makes an FADEC tearing a helicopter apart look like peanuts when they couldn't get a simple dedent to work as they intended. (There isn't even a single piece of electronics in the dedent mechanism itself and you can't have BUGs in a mechanical part. They knew it was out of spec, they knew it caused deaths, and they knew it needed to be recalled.)(Toyota would be commiting many countless crimes if they covered up the ECU fault and juding by how their internal emails are floating around I'm sure they were not aware of any catastrophic bugs if they existed or not)

Show me actual EDR statistics pointing the finger not at the driver because the self reported numbers if you know anything about statistics are affected by more than just it being unreliable (The media factors hugely into those self reports, hard statistics needs all the variables controlled for and an EDR report is consistant, quantitative, and objective) The self reporting system is an indication not proof.

You should accept the possiblity that an expert witness hired by the platiff sueing Toyota for millions in damages may be slightly baised to say the least. The DoT and NASA have no interest in Toyota's sucess arugably as a forign company they would like GM (also partly government owned) to suceed and maybe thats why the national highway regulators didn't make more of a fuss about some really strange EDR reports with no air bag deployements and complaints about stalling cars going from on to accessory or even off.

Also it is clear that your ignoring the differences that I directly relate to this article and countinue to ride on how similar GM's mechanical defect is to Toyota's supposed software defect I have no choice conclude that you have placed your crediblity and worth in the hands of another unknown expert based on unpublished reports and unverified testing.

I neither believe that Toyota, GM, Boeing, Airbus, any programmer can write bug free code and that they all hopefully tried as hard as they code to write safe code within the best of their ablities and that in Toyota's situation real world statistics appear to favour their opinion that it is drivers at fault while in GM's case it is clear that it is not the driver's at fault as keeping keys on a keychain isn't something the manual says not to do. (The GM situation has nothing to do with electronics and everything to do with managment/engineer/supply chain failure)

Attempting to refer to Toyota's supposed overly complex ECU bugs isn't remotely similar to GM's known non-supposed simple glaringly obvious mechanical defect in a simple part that went unrecalled for 10 years and had many deaths/injuries directly attributed to the fault to date in real world statistics and EDR records.

 

 

 

Answer this, GM has EDR reports that point the finger directly at the cause and Toyota doesn't, why are they both on the same level of coverup?

Assuming your right and the ECU buggy mess it doesn't cause errors then Toyota/Regulators wouldn't have known and the difference in the article is that GM knew it was clear as day for 10 years. (Silently fixing something and not fixing every affected car can't be considered "problem solved")

 

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 10:33:39 PM
NO RATINGS
Want to know about a real "ECU" software problem and what happens even if the programmer denies it?.

http://www.computerweekly.com/news/1280091852/Faulty-safety-critical-software-changed-after-fatal-Mull-crash

FADEC (basically a jet engine equivlent to an ECU) with horrible programming to the point where even beta testing showed it to be totally unfit for use and cause total engine destruction and possibly many other failures. The catagory 1 defects in the program where likely totally obvious and known oversights which just where not fixed. Having code which looks bad and an outside "expert" claims in unmaintanable is not the same thing as a clear software fault that even in production without any alterations or debuggers which causes an engine problems in testing being called unintelligable and using/relying on undocumented hardware features.

(Arugably having a single main CPU and smaller monitor CPU would have been better for that programmer as the monitor CPU has very simple tasks to carry out and even if they totally botch the main CPU task X they are not running two copies of it or don't have to try and program two different giant task X's but one large one and a tiny task Y) The programmers could be a failure point as well and a pretty catastrophic one in that helicopter's case.

In that case the FADEC programmers were making code so horrible it was using features that where not even supposed to be used on the CPU so they were in essense making the hardware run out of spec which is something that Toyota hasn't appeared to do from the testimony even as that would be a pretty glaring safety failure if they did do that but they used two different CPUs and appear to have sound safety design for a ground car application. Code review in the helicopter case said it was totally unfit and when pushed into service the helicopter crashed and the software was changed and the out of spec CPU as well and they went on. Toyota hasn't recalled and no actually documented cases have been reported so no catastrophic software fault is assumed to exist.

GM has a known failure with a known fix with a long time delay and a non-recall warning to dealers which only now is being addressed years after the cars went out of production and under new managment. GM had a total systemic engineering to managment failure while Toyota had a PR disaster which was technically not their own doing (Maybe they could have wording things better than saying those dead drivers basically killed themselves for the most part but techically they are right in the bulk of the cases). (GM on the other hand has people dying and crashing not because of driver error but a simple mechanical problem with nothing less than a mechanical switch dedent being under spec)

rich.pell
User Rank
Blogger
Re: Like Toyota?
rich.pell   3/8/2014 9:10:45 AM
NO RATINGS
"I see no way around the apparent fact that Toyota's Task X can crash, fail to reboot, and cause the throttle signal to show full open."

Showing that something is possible is not the same as proving that it actually happened. What is the probability of this "apparent fact" happening under normal driving conditions?  As this is a technical matter, the probability of its occurrence should be able to be determined by simulation or actual testing. Has anyone provided such data?     

"I agree that under such circumstances ... a totally alert driver *should* be able to stop the car."

The drivers in all of these cases claim to have been fully alert and pressing fully down on the brake pedal. 

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/9/2014 6:12:57 PM
NO RATINGS
Showing that something is possible is not the same as proving that it actually happened.

While that statement might be true, it's beside the point. When designing anything that is supposed to be safety critical, you cannot allow known mechanisms of failure to go unaddressed. That's what "fail safe" means, after all. This holds true for GM and for Toyota. Most of the time, you have to demonstrate that it takes at least n simultaneous failures before the unsafe state can exist. Very standard engineering practice.

The drivers in all of these cases claim to have been fully alert and pressing fully down on the brake pedal.

True. And sometimes, due to the positioning of the pedals, it was determined that they actually had their foot planted firmly on the throttle and not the brakes (this applies to Audi back in the 1980s). Other times, such as in the Toyota case, it was instead quite likely that the brakes were not working, or that the vacuum assist was not (due to the full throttle command), and that therefore the driver needed to apply a whole lot more brake pedal pressure than they normally would. Such an event could easily be construed as "the brakes were not working."

The basic point remains, though. Faced with an extraordinary set of circumstances, many, if not most, human drivers do not react properly. So while it might be true that the driver "should have" done this or that, the reality is that many or most drivers don't.

I feel no compulsion to take sides with or against Toyota or GM, in these cases.

 

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/10/2014 4:55:45 AM
NO RATINGS
Your post is patently false.

Real hard statistical reliablity in real world testing will always trump theoricial reliablity (Saying it might be unreliable when it isn't really isn't a factual statement). If you say a part can only last for 10 hours in use due to advanced FEA modeling showing a clear defect in the design but 100% of your customers used it for 10+ years ignoring your unproven model and no failures or even indications of failure occurring which one is correct. Debuggers inducing a percieved bug doesn't prove a bug exists as already mentioned without access to the technical details of their test and other groups not reproducing the same results it can only be concluded that your theory is wrong.

Vaccum assist works even at full thorttle it just won't recharge. If a user claims they pressed full down on the break then the car will stop with minimal effort and a forceful stop can occur. Even if they don't read the manual or understand basic car tech a depleted vac assist doesn't stop the user from slowing the car down safely and a runaway condition only occurs if they don't do anything or are actually pressing on the accelerator. It has already been proven that the pedal sensor for the brakes show that it wasn't depressed in the EDR reports so nothing was even trying to stop the car and the braking system is not even electrical (the ECU has nothing to do with it and is another layer of protection on top of a number of other simple defenses against a rouge ECU that hasn't even been proven publicly) The EDR records brake pedal position and almost all cases had them slamming hard on the accelerator (pedal misapplication).

The most basic point remains in that the unpublished, not-repeated, unverified "expert" theory on what happened did not happen in real world situations. Drivers have a responisblity to know how to turn off their car, use the neutral, and press (not pump) the brakes when needed, and of utmost importance in not mixing up the pedals (Pilots of planes are expected to know what to do if the autopilot disconnects and that doesn't involve entering a unrecoverable stall by pitching up maxiumally when the clear thing all pilots should know is to pitch slightly down to exit a stall regardless of lost or confusing sensor readings gravity/aerodynamics basics don't change).

Experienced drivers can use both pedals at once and brake override is a "noob" feature which doesn't even help if the driver doesn't mash both pedals which as shown in actual EDRs is almost always just them slamming the accelerator to the floor and a pedal override wouldn't help. (it is just to platacte people).

Your bias regardless of what you claim is so apparent it is surpising you need to state that as the poster your replying to never implyed any of that, who here ever claimed your biased to GM or Toyota other than yourself now. I don't care about Toyota and I don't own stock or even own one of their cars I just find it appaling that according to your profile pic a Principal Engineer at Boeing Defense Systems thinks a supposed ECU bug is somehow similar to a switch detent being off spec. (linked in is a bit creepy I must say but its in public space so fair game)

Care to chip into to how you think Toyota's ECU is similar/different to the Beoing Chinook disaster. http://www.ccsr.cse.dmu.ac.uk/resources/general/ethicol/Ecv12no2.pdf

Because I think that case the first thing I searched up on FADECs (jet engine eq, ECUs) was that and I was surprised that boeing would ignore their own internal testing resulting in some catastrophic damage to the test helicopter's due to obvious control system bugs (no monitor CPU on the sensor lines to idle the throttle control so it instead went to 100% and would have and did later self destruct the entire drive train and frame in flight). I dunno but when the CPU arch is switched entirely and obviously a total rewrite occured it is pretty clear which case had bugs and which didn't. (Toyota never swaped the ECUs out and no recall and no further cases and there were likely none to start with in regards to an ECU bug).

 

Get me the actual report and then I will consider it but making random points without backing it up with any information isn't proving anything. Induce the bug in an actual car without a debugger and prove it publicly on tape with EDR. Compared to the miles driven the probablity of this supposed bug is slim to none and we don't even know what it is that they are changing in the debugger or what the task is called. It definitely isn't an app I can tell you that much it probably is the entire ECU program and in C.

http://www.embedded.com/electronics-blogs/best-practices-in-programming/4219722/Don-t-follow-these-five-dangerous-coding-standard-rules

Interesting comments from him when he suggests using a division operator when the mfg specifications specifically states not to do it and the resulting compiler madess to accomodate something like a divide on a platform without a division or multipler results in horribly unoptimized code. Using maximum sized variables is a waste of space especially on highly RAM constrained systems, sure on a x86 or ARM processor don't even bother doing hand optimization but on a tiny 8 bit memory/flash/cpu starved micro not doing hand optimization and assembly tricks and just letting the compiler deal with it is not going to work. He ignores 95% of the embedded systems out there by suggesting readiablity above all else as many products can't even afford to add extra resources just to keep readable code around (C++ on an 8bit?, C# on an 8bit???, python?, .......) In all those it makes sense not to attempt hand tweaking (not that intel IPPs don't do that in spades).


Barr's wikipeida article reads like an ad and the cross editing is a tad suspecious and I sure hope all those editors are revealing their potential conflicts of interest before modifying a public encylopedia add paragraphs in every UA article about how the ECU is buggy. Strange that they haven't pounced on FADECs.

 

 

 

 

rich.pell
User Rank
Blogger
Re: Like Toyota?
rich.pell   3/10/2014 10:12:07 AM
NO RATINGS
"When designing anything that is supposed to be safety critical, you cannot allow known mechanisms of failure to go unaddressed."

In the Toyota case we are not talking about a "known mechanism of failure" but a hypothetical failure mechanism that has yet to be demonstrated to have occurred in reality or even assigned any probability of its occurrence.  

"Statistically, it was shown that the Toyota unintended acceleration problem was far worse than occurrences from other companies."

Some of this can be attributed to the floor mat issue or pedal positioning, as you point out.  And, no surprise, Toyota UA-related complaints shot up after the massive publicity about the issue a few years ago.  So the statistical data does not appear to be nearly so cut and dried:

Unintended Acceleration In Toyotas: The Ghost In The Data

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/10/2014 4:02:22 PM
NO RATINGS
In the Toyota case we are not talking about a "known mechanism of failure" but a hypothetical failure mechanism that has yet to be demonstrated to have occurred in reality

I'm not sure I even understand why insist on this.

This same sort of thing happened to me. I wrote a control system algorithm that was supposed to work in a particular way, to be able to withstand two simultaneous failures. During testing, I noticed that it wasn't implemented the way it was meant to be.

Were there any known failures in the field? No, not yet anyway. Did I just forget about it? Of course not. How ridiculous would that have been? I wrote up the problem, its potential effects, what it would take to fix it, and passed that up the chain.

It's entirely possible that the floor mat problem masked other problems. Pedal positioning was not mentioned as a possible issue in this case, so it makes no sense to invent an extra mechanism of failure. A "hypothetical failure mechanism" that is known to exist must be addressed. If you can show that the "hypotehtical" actually CANNOT occur, that would be a different matter.

rich.pell
User Rank
Blogger
Re: Like Toyota?
rich.pell   3/11/2014 11:47:50 AM
NO RATINGS
"A "hypothetical failure mechanism" that is known to exist must be addressed."

There is little or no evidence to suggest that Toyota engineers knew of any such failure mechanism and failed to address it.  A team of NASA engineers tasked with finding such a problem found no such failure mechanism. An objective look at the statistical data - adjusted to correct for the floor mat problem and the "publicity effect" - does not show Toyota having a significant problem in this area relative to that of other auto manufacturers.  However, an expert witness paid to go through the tens of thousands of lines of code with a fine-tooth comb to find a problem - lo and behold - found a "problem" and managed to convince a (presumably) non-technical jury that Toyota was at fault in one case.  Certainly if this is a theoretically possible issue that can be addressed, I would expect Toyota would update their software accordingly. But given the weight of the evidence so far, it does not seem that Toyota deserves the excoriation it has received here in articles and threads on this topic.

"If you can show that the "hypothetical" actually CANNOT occur, that would be a different matter."

Of course.  But the burden of proof is on those making the claim.  Proving a negative is likely to be far more difficult - if not impossible - than actually proving the initial claim.

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/11/2014 3:55:14 PM
NO RATINGS
There is little or no evidence to suggest that Toyota engineers knew of any such failure mechanism and failed to address it.

We can certainly agree on that. This is not to say that the problem may not exist, however. If the problem is bogus, then Toyota is free to counter-sue, after showing why the description by Barr was bogus. We have no reason to believe that Barr was spreading falsehoods, as of now.

However, an expert witness paid to go through the tens of thousands of lines of code with a fine-tooth comb to find a problem - lo and behold - found a "problem" and managed to convince a (presumably) non-technical jury that Toyota was at fault in one case.

Correct. And since unintented acceleration reports for Toyotas were statistically larger than from other auto companies, it is certainly not beyond belief that this problem is real. I wouldn't be a bit surprised if Toyota has taken this input seriously, to determine whether the problem is verifiable. That's what I would do, for sure. It's very possible, with thousands of lines of code, that something was missed back when. Previous attempts at bombarding these cars did not look in depth at the code itself, just the effects. A fault in the logic that only rears its ugly head rarely can certainly have been missed.

Again, it is beyond me why we are even going on and on about this. When safety critical mechanisms are being questioned, it's inconceivable to me that anyone would gladly ignore the problem. You have to show that the logic is correct as is, or you develop a fix. It cannot be so difficult to show that task X death cannot possibly create a false throttle command.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 8:20:47 PM
NO RATINGS
Yeah, using a specialized software debugger to cause an UA isn't exactly proof of anything. Similar to the other "experiement" by removing the ECU and overriding the engine control electrically just using a debugger to alter the ECU in software isn't proof of anything as in debug mode the ECU won't function the same as in production as debuggers induce their own effects especially on installed hardware controlling a real time system.

Also they demonstrated their lack of expert knowledge by the very thing I mentioned before, "Does it make sense that you're braking but you are having to fight the throttle because it is open 50 " (Hills that's the answer, steep hills, you want the two to fight but I guess jerkly moving up the hill or just loading that transmission up is better I guess, doesn't matter all evidence is on the drivers fault for the most part) A debugger induced UA isn't proof and so much [redacted] it doesn't say much at all other than its their fault software fail no reason other than big software fail.

They didn't mention how the sub-CPU was bypassed or why it wasn't involved or what the debugger does to the ECU when connected and how it might affect results. Nor does there have to be a DTC to show strange data in the EDR. (Which records all the sensor values so unless you have a transcript of them showning an EDR report it never happened and all the EDR's pulled from accidents showed "pedal misapplication" as the primary cause)

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 8:27:15 PM
NO RATINGS
I'll agree that there was a lot redacted, so we aren't getting the complete story. Still, I think that what happens after the case resumes will tell more of the story. If I were the software designer, and a flaw was discovered in my software, even if the risk is very low, I'd certainly want to change it!

Oh yeah. Now that I think of it, I've that happen to me too.

adornao
User Rank
Rookie
Re: Like Toyota?
adornao   3/7/2014 8:50:25 PM
NO RATINGS
The problem with your thought on software is that it is virtually impossible to eliminate bugs from even safety critical software. Designing the software and hardware such that you can be as lazy as possible in the main CPU and lazy in the subCPU and have electrical/mechanical failures without causing death/injury is good design. Relying on good code relies on a select number of human programmers to be really good at what they do and that is a point of failure that could easily occur. Writing different pieces of software, getting electrical designers to chip in, and adding a mechanical engineer to design the brake/engine power balance makes it so that no one design failure could cause a problem.

In Toyota's case there doesn't appear to be any problem with any level of design short of jamming a debugger and commanding what is probably a throttle override routine which the fail-safe's are probably disabled when a debugger is attached as the ECU would be in a test-jig/pre-production mode where full throttle testing would be desirable and attempting to override it by physical inputs is unexpected and admirable that the car could even stop with a debugger bit flipping a select [redacted] value. This entire previous paragraph is conjecture because they don't explain things clearly and no sound technical details where given.


For a Toyota car to crash the user has to one press on the wrong pedal (or something that never has been recorded in practice happens and it looks like full pedal) and two not know what neutral means, not know how to use brakes properly, and not know how to turn off a car. (GM's got that in spades I guess)

Toyota is basically a human factors case plain and simple, GM's is a total design and subsequent handling failure.

And obviously I'm not saying write horrible code as that will cause failures but it is going to happen (doubtly it caused it in Toyota's case) and other design aspect can mitigate it and did in Toyota's just the user's didn't know any of the layers of mitigation steps they could have done. Even in the unfortunate improper footmat speed/firey crash they could have just crashed into the median to stop themselves and that would have been far better than flying into a ravine (Hindsight 20/20 yes but maybe some better driver training on car basics and emergencies would help, I read a story where an engineer used a controlled car crash to help another person, search "Driver thanks man who hit him on purpose")

Training 911 and drivers on what to do would be far more effective and reliable than adding quad redudant/diverse os/physically seperated ECU control. Sometimes the solution/problem isn't electrical.

And again obviously if they knew about a bug they should fix it but unlike GM's case where they did know Toyota doesn't seem to think there is an issue and is there a multi-million car recall to reflash the firmware or swap the ECUs (no there isn't) only ABS software bugs in suboptimal breaking distances (Old ABS software is far worse arugably), and mechanical issues where addressed.


Finally the biggest nail in the unintended acceleration cases is that pressing the brakes and throttle does stop the car and numerous news agencies tested it even without brake override. (Some claiming it didn't but their claims are dobious at best)

 

krisi
User Rank
CEO
Re: Like Toyota?
krisi   3/7/2014 7:38:50 PM
NO RATINGS
I think GM is at fault, they clearly were dragging their feet...not sure why Junko thinks it should be regulator role to oversee this...car companies should be responsible for their products

Bert22306
User Rank
CEO
Re: Like Toyota?
Bert22306   3/7/2014 7:45:41 PM
NO RATINGS
I agree Kris. And I've also manually tested the torque on my ignition switch!

DrQuine
User Rank
CEO
how were ignition switch issues missed?
DrQuine   3/4/2014 6:16:59 PM
NO RATINGS
Hindsight is always 20-20 but some issues are hard to uncover at first. I suspect there were three issues at play here. First, was there a torque specification to move the ignition switch between positions? If not, there there was no test for incoming quality assurance to perform to ensure that the switches met torque specifications. Second, as a switch ages, the components may loosen up and the necessary torque may be reduced. If the torque parameter is not part of the specification, then there would be no reason for repetitive testing of the switch to see if the necessary torque reduced through time. (The expected lifecycle testing to ensure the switch continued to work would not detect this issue). Finally, there has been a trend for keychains to get heavier and bulkier as remote controls, multiple car keys, house keys, work keys, and additional security features get added to the mix. A single key in the ignition could probably run forever without causing trouble. With the benefit of hindsight, it all makes sense. The necessary specifications and test protocols should be simple to implement.

DrQuine
User Rank
CEO
Additional Factors To Consider In Ignition Switch Designs
DrQuine   4/2/2014 4:56:47 PM
NO RATINGS

The light torque required to turn off the GM ignitions and the drivers' heavy keychains have been widely discussed in the GM car crashes. What has not been discussed is the angle of the key in the ignition and the shape of the keyring holes in the keys.  These two factors should also be considered to prevent accidents.

[A] IF the key has a long keyring slot across the head of the key, then if the key chain slides to the edge it may have additional leverage on the keyhead and increase the potential torque on the key to turn off the engine. [B] IF the keyhead is vertically oriented when the car is driving, the keychain may help hold the key in the "ON" position whereas if the key head is horizontal when the car is driving, the weight of the keychain may tend to pull the key into the "OFF" position.

These factors should be considered in future ignition configurations to avoid designs which are at risk of failure.



DrQuine
User Rank
CEO
Confirmed - Additional Factors To Consider In Ignition Switch Designs
DrQuine   4/5/2014 9:54:22 PM
NO RATINGS
My prediction three days ago appears to fit the facts! I just saw a news report in which the GM key in the ignition had a long slot across the head which enabled the other keys hanging on one side of the key to create the torque to turn the key from the "ON" [horizontal] position to the "ACC" position in which the key head is vertical.



EE Life
Frankenstein's Fix, Teardowns, Sideshows, Design Contests, Reader Content & More
Glen Chenier

Engineers Solve Analog/Digital Problem, Invent Creative Expletives
Glen Chenier
3 comments
An analog engineer and a digital engineer join forces, use their respective skills, and pull a few bunnies out of a hat to troubleshoot a system with which they are completely unfamiliar. ...

Max Maxfield

What's the Best Traveling Toolkit?
Max Maxfield
22 comments
A few years ago at a family Christmas party, I won a pocket knife as part of a "Dirty Santa" game. This little scamp was a Buck 730 X-Tract. In addition to an incredibly strong and sharp ...

Rishabh N. Mahajani, High School Senior and Future Engineer

Future Engineers: Don’t 'Trip Up' on Your College Road Trip
Rishabh N. Mahajani, High School Senior and Future Engineer
10 comments
A future engineer shares his impressions of a recent tour of top schools and offers advice on making the most of the time-honored tradition of the college road trip.

Larry Desjardin

Engineers Should Study Finance: 5 Reasons Why
Larry Desjardin
42 comments
I'm a big proponent of engineers learning financial basics. Why? Because engineers are making decisions all the time, in multiple ways. Having a good financial understanding guides these ...

Top Comments of the Week
Flash Poll
Like Us on Facebook
EE Times on Twitter
EE Times Twitter Feed

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)