Breaking News
Comments
You must login to participate in this chat. Please login.

There's been a lot of reporting on the testimony of Michael Barr who testified for the plaintiff. Did Toyota field their own expert witnesses ? If they did, it would be interesting to learn what their rebuttal was.

Rookie

The undoubted fact that all of us who drive make mistakes should not be used by the automobile companies as a cover for ignoring functional safety issues and making the driver the fail safe for malfunctioning electronics. What has come out of the woodwork through Dr Barrs testimony is that the EDR results may get corrupted and fault codes disappear when Task X dies - so the software has perfected a way of exonerating itself and the automakers and allowing drivers to shoulder the blame. This might not matter so much if it was not for the fact that people are getting injured and killed and blamed for it and insome cases, if they survive, ending up with lengthy prison sentences. At the end of the day there are moral issues here and  public safety. 

Yeah, software-driven cars need to be more reliable than human-driven ;-)

Manager

There's tons of chatter out there on various message boards & sites about this case. People are clearly interested.

Manager

The most dangerous element is still... human. I'm in a cab and somoene just swerved in front of us and almost took the cab's fender off. I'd like to update that guy's software...

Thanks everyone. It is wonderful to see so many people interested in the issues and putting in ideas. May this be the start of something bigger and wider

Thank you Betsy, Antony, Mike...!

Manager

No matter if the Toyota Camry problem was due to penny-pinching, bad design, shoddy design, inability to anticipate where problems might probably occur, the fact is modern automobiles are  systems far more complex than the Model T or Model A and even the best design will have interactions than cannot be anticipated, even assuming a rational driver who is knowledgeable about driving and about the tool – his or her vehicle – and what to do when the unanticipated occurs.

But in most countries, except maybe in Europe, there is little or no education. Unlike the situation in the latter part of the last century, In the US, there is no requirement for driver education in school and the driver's test you take to get a driver's lisence is micky mouse. And as for knowledge about the tool – the inherently dangerous tool under most normal driving conditions – forget it.

Rookie

Not just software, but look at the hardware and systems. Dont forget the physics of failure approach as followed by Michael Pecht at CALCE

Right, Michael, that too....Once again, thank you all for chiming in

Blogger

Thanks to all, and especially our host, Junko.

Blogger

I have long expected that as automakers 'relaxed' their testing by not vetting such complex issues as software, and by accelerating the introduction of more and more technology at lower cost, that a decision would have to be made:

Do we increase cost and slow innovation by testing to what we suggest here to be FAA standards?

Increase cost and weight with backup/redundant systems?

Do we ask buyers to sign a disclaimer?

Do we confine rapid innovation to non-critical systems and fall back to robust, proven mechanical systems for acceleration/braking?

I do like Colin's suggesetion of a big red "STOP" button. If we can have that on a treadmill, why not a car?

Thank you for hosting this Junko Yoshida

Rookie

We need to keep asking what they will do next

Blogger

@ dabebarman.

It doesn't have to be an official "bug" to kill, just an algorithm that didn't take a particular set of circumstances into account!

...isn't that just a bug in the model. As systems become more complex, the models do too.

Freelancer

@bsnguy - Nike's "Just do it" is quoting a doomed convict's last words before he was executed.

Manager

conclusions include: NHTSA, IEEE, and the auto industry

Blogger

Definitely Mike. System design is where it's gotta start. A good system design can mitigate some crappy sub-components. Not that there should be any, but, things fail...

Manager

but there are a lot of good nuggets here

Blogger

This feels like only the beginning of the great discussion, but I am afraid time is up

Blogger

"we do fly-by-wire planes, in the presence of increased radiation at higher altitudes. I believe we can handle the complexity of cars"

 

Absolutely! As Nike would say: Just do it!

Rookie

We are talking of ensuring the stability of of interacting vehicles each with its own power supply. So ensuring stability is bound to be a major problem - where and how is the multi-vehicle system to be kept stable?

one comment on complexity -- we do fly-by-wire planes, in the presence of increased radiation at higher altitudes. I believe we can handle the complexity of cars

Rookie

@embeddedbarr, that system-level choice is definitely something we can all chew on

Blogger

@Tom: as a former Intel employee my thots at the conoclusion of a meeting are - what's the follow up action to be taken?

 

Great question. Is anybody from the auto industry listening? How do we get them to understand that electronic systems relability is important and they need to pay attention? I thikn they are due to the Toyota case, but are they looking at the right things (or are they focused on "software" as the only problem/solution)?

Rookie

@Hess, I doubt Michael can answer. I think it was the Toyota attorneys who demanded he take it down.

Manager
dabeberman I think a more fundamental problem with autonomous driving may be proving the algorithms won't suddenly make a life-ending decision incorrectly, but not because of a software bug, just the way the algorithms work.
 
Good point. It doesn't have to be an official "bug" to kill, just an algorithm that didn't take a particular set of circumstances into account!
 
Blogger

@junko would differ with the statement about the financial industry....

Rookie

 I noticed that Michael Barr's account of the Oklahoma trial and the link to the transcript have disappeared from the web - any ideas?

Rookie

Keith Armstrong of Cherry Clough Consultants and EMI expert talks of the "thousand car pile up".

Yes, we all know driving is dangerous. And that cars have gotten safer. But injury/death caused by poor & incompetent design. That's unacceptale IMHO.

Manager

The systems are very complex. But there are methods for dealing with it. The problem is that these methods have inputs that are not being supplied, today.

Rookie

@tom, it was the same issue that caused the massive financial failure -- things got so  complex that nobody can understand 

Blogger

 I think a more fundamental problem with autonomous driving may be proving the algorithms won't suddenly make a life-ending decision incorrectly, but not because of a software bug, just the way the algorithms work.

Rookie

as a former Intel employee my thots at the conoclusion of a meeting are - what's the follow up action to be taken?

Blogger

But what are you certifying against? I would say that certification should be that you have demonstrated that your systems are immune to radiation hazards to some level; that you have mitigated metstability failures to some small level; and so on... but if we don't know what those possible failure modes are, and/or if we don't analyze them, we can't define the standard in the first place.

If the auto industry wants to take the lead, here, they could simply start doing the analysis and then they can write the standards! Yes, they can! And I'd be happy with that if they prove they have done the analysis and that they understand the results and that the standards they set forth are reasonable. Oh, and there needs to  be a public review of all of this.

Rookie

Autonomous vehicles - remember the stability problems that we have controlling a large generation and transmisstion network. Think of the equivalent brownout with a traffic network.

CAN networking was grewat for the auto industry to reduce the wiring complexity. It is not a relaible (physical) transport, and I doubt can be made reliable enough for the sensor network.

It's ok to turn lights on and off, thats about all.

 

Freelancer

there is an over-arching issue here.  have our systems - in cars, health care, financial markets... - become so complex they are unmanageable  except by even more complex systems which in turn require...?  where does creative destruction end?

Blogger

>@dabeberman thought someone has already shown that you can hack the CAN through the tire sensor or something like that

I think you are right, but no verified crashed caused by hacker yet--I don't think. Anybody know of verified hacker caused crashes?

Blogger

>>But, a failure may now cause more damage (or death) so the numbers of deaths/injuries per failure may be higher.

Are there any statistics to support this?  I suspect cars are also probably safer than ever as well.

 

Blogger

JPL has some videos on the automotive issues from flight software workshop KEYNOTE (Day 1): Software Development for Safety-Related Automotive Systems
Dr. David Ward, MIRA

Blogger

@Tom and @JC: Agreed. Complixity is increasing. We are just at the start of the new reliability issues.

Author

JCreasey, unless, every car on the road is autonomous

Blogger

@B.Benjamin get Consumer Reports to write an article on the value of independent 3rd party validation and verification certification for autos

Rookie

The most eye-opening thing I've learned is that "neutral" is only a suggestion to the software that may be malfunctioning.

 
 

 

Scary. I still drive a 1998, and am unaware of these early 21st c cars you speak of...

Manager

Two wire system - dont forget also that the vehicle is used as ground return. So there can be plenty of problems of ground lift e

@Tom Mahon. It it the tip of the iceberg....as we move to autonomous systems the problems apparent here with multiply exponentially.

Imagine thousands of autonomous vehicles interacting (with faults)

Freelancer

So to get third-party certification of auto electronics safety, you could go with governments requiring it or consumers being taught to demand it. The UL might lead such a consumer driven effort.

Manager

@R_Colin_Johnson -- thought someone has already shown that you can hack the CAN through the tire sensor or something like that

Rookie

I suspect that cars ARE more reliable. But, a failure may now cause more damage (or death) so the numbers of deaths/injuries per failure may be higher. We need to mitigate failures so they are not catastrophic.

Rookie

I can recommend that you take note of Ron Belt's work on the possible part played by the battery.

 

Suggest you google "BELT HYPOTHESIS SUDDEN ACCELERATION" and you should find the relevant page on my website where the pdf files can be found. If not, e mail me

@Tom Mahon - as others have stated, this is not new territory, just new to the auto industry. We do know how to mitigate a lot of went wrong here, to do some level of risk mitigation

Rookie

>two wire system, too easy to use, even though it is not secure

Yes, it will take verified hacker-caused crashes to change from CAN

Blogger

this exchange has been a real eye-opener for me in terms of the universality of this issue.  we all have our favorite stories of problems resulting when 'the system crashed.'  but when working with a desktop or laptop that was an inconvenience.  system crashes at 60 mph are a whole other universe of pain.

Blogger

@B.Benjamin -- agree with 3rd party verification houses

Rookie

>> Anyone seen any stats on the complexity of the average car today and the statistical probabilities of failures?

I suspect cars are probably more reliable today than ever, despite being so much more complex.

Blogger

@junko ...agree that IEEE is the place to start, but you have to have teeth to the standards to ensure implementation worldwide.

Freelancer

@Antony Anderson -- the industrial automation space is going to Profinet. Doesn't look like the automotive space is going to move away from CAN -- two wire system, too easy to use, even though it is not secure.

Rookie

@bsnguy Ford MS Sync sits on the Canbus with the other 50 processors....


Corrected... so it's possible that an event from this system could casuse a signal to get onto the bus that was unexpected and cause another system to fail. Great... I feel so much better!

 

Rookie

>@R_Colin ....That would be great...what does it disconnect. I'd love to be doing 60mph and simply turn of the drive by wire functionality.

>If we are going to go electronic, then the sytems have to be reliable.

Of course you are right. I should have said - tongue in cheek :)

Blogger

@dabeberman, I am not sure if we need another standards body... but IEEE is the first place I would talk to

Blogger

Third-party safety certification is not being done for autos. Really it must be done.

Otherwise, a company like Toyota can trumpet its "redundancy" in PR and marketing, but that may actually mean they put two sensors on one chip. Which was the case here, in the gas pedal position sensors.

Manager

The CAN bus is another area that needs looking at.

@junko so we start another standards body up?  Get the experts together, look for harmonization internationally?  Could work.

Rookie

@R_Colin ....That would be great...what does it disconnect. I'd love to be doing 60mph and simply turn of the drive by wire functionality.

If we are going to go electronic, then the sytems have to be reliable.

 

Freelancer

Certifications and regualtions are all "after-the-fact" solutions. We need to understand the fundamental issues before we can regulate them or cause people to be certified to meet some bar. I've seen little discussion of the underlying problems and that's what is bothering me. All of this talk about software makes it sound like there is an easy fix (for everyone knows that software is easy... of course it's not, but that's the conception). I suspect there is no easy fix; the systems are simply too complex so all of the handwaving that used to work will now fail. We need to get back to fundamentals and understand (then mitigate) failure modes.

 

Rookie

>>already a lot of people died in the Toyota case

 From what cause has not been proved.  People die in car accidents all the time...

Blogger

@Colin, someone in the forum also suggested big red button on the dash. But it won't work when the vehicle is moving 

Blogger

 It might be a good start if the automobile industry recognized that Murphy's Law - whatever can go wrong will - applied to electronics. At present their method of dealing with electronic failures seems to be to blame the driver. Witness the effort to "prove" every incident of sudden acceleration to be due to "pedal error". If they had put in a totally independent fail safe when introducing the electronic throttle we would not be having this discussion now.

@B.Benjamin, already a lot of people died in the Toyota case

Blogger

@bsnguy Ford MS Sync sits on the Canbus with the other 50 processors....

Rookie

Neither Windows or Linux are viable in any life value situtation, but why would you want a large multitasking OS for tasks such as this?

 

Freelancer

>Redundancy is the only way to ensure ultimate reliablity.

How about a big red "Shutdown" button in the middle of the dash?

Blogger

The problems with waiting until manufacturers act differently due to fear of litigation is--first, someone has to die first. Second, the cost of the biggest court loss is chump change.

Manager

@dabeberman, I am not sure about world wide

 

Blogger

@Michael Dunn -- !!!! please do not promote Linux for safety systems.  

Rookie

as far as NHTSA is concerned

Blogger

I'm not worried about Ford MS Sync. If it failed, my car should still run (well, maybe I'm distracted by it, but that's another issue). I'm concernd about the 50 or so procesors controlling everying from my engine timing to the breaks to the airbags to the steering. Oh, and they all communicate so I'm worried about that, too...

Rookie

@junko - so right now, there are no safety certifications required for automotive controls software, world wide?

Rookie

Linux is certainly making auto inroads. But NOT in safety systems! Just in "infotainment"

Manager

I think the basic problem for the auto industry is that as systems become more complex the number of single points of failure increses, and they have never implemented any forms fo real redundancy.

Redundancy is the only way to ensure ultimate reliablity.

Freelancer

>would anyone want to drive in a car running under Windows?

Windows is a perfect example of software that was engineered to just work right doing things "one way" rather than testing to make sure all ways are "safe"

Blogger

I have seen a document in which there was a scribbled matrix of the costs of various types of recalls. And evidence of efforts made to limit recalls.

And one email expressing hearty congratulations for limiting a 2007 recall to floor mats. The writer was a little incredulous that they succeeded in getting NHTSA to pin it on floor mats.

Manager

@Tom Mahon -- Ford MS Sync?  source of latest complaints by Consumer Reports - how long before it is blamed for an accident?

Rookie

>reduce them to reasonable limits

Key word in bold. While engineers think of reasonable limits, the probability of an occurrance, lawyers don't. It only takes one instance to cast doubt.

 

Agreed! And if the laywers get invovled, that's good in this case! It will force the manufactures to do the analysis both to improve reliabilty AND to guard against lawsuits. "Your honor, we did everything humanly possible to prevent such an event." If thay cannot say that, I would hope they would change their ways so they could.

 

Rookie

Re cost, there are the infamous stories of mfrs basing a recall decision on cost of recall vs. cost of lawsuits...true or not.

Manager

Anyone seen any stats on the complexity of the average car today and the statistical probabilities of failures? Like hard drive mean times between failures?

Author

would anyone want to drive in a car running under Windows?

Blogger

IEEE automotive controls - http://www.ieeecss.org/technical-activities/automotive-controls

Rookie

@dabeberman, true. But then, from what I understand, they've got nothing right now...

Blogger

>>reduce them to reasonable limits

>Key word in bold...It only takes one instance to cast doubt.

It only takes one instance to kill. Do we need some kind of "panic" button or manual override?

Blogger

A company called Quality Control Systems in Md. is keeping track of current Toyota SUA complaints to NHTSA. These complaints are posted on the company's website. The numbers have come down somewhat, but still quite significant, even in new model years. This jives with one of the Toyota documents in which an executive expresses concern in 2010 that the poor quality practices are actually "proliferating." So they are not over.

Manager

@junko.yoshida althought I'm for safety-critical standards. There is a tremendous cost associated with the FAA standards, and it significantly slows down development.  Think 2x to 4x at least in extended time.  The question will come up, does the automobile need this level of certification.

Rookie

There are reliability engineers working at every auto manufacturer. But they don't have the data they need to do the job. What can fail? What's the probability of failure? What are the modes of failure? There are tools to do this analysis, but I don't know of any auto manufacturers using them. Things like I mentioned, before: radiation, metastability and other forms of failure can be characterized, but are they? I think not.

 

Rookie

>reduce them to reasonable limits

Key word in bold. While engineers think of reasonable limits, the probability of an occurrance, lawyers don't. It only takes one instance to cast doubt.

Yes the IEEE and the IET and various other professional institutions should be involved because there are major issues of public safety here.

@dabeberman, there is definitely that -- penny pinching

Blogger

>For the engineers out there (I am one) we need to understand the underlying issues that can cause failures and work to reduce them to reasonable limits.

Good advise! Too much emphasis is on delivering by the due date over understanding the underlying issues.

Blogger

Since the NASA study, NHTSA has established an electronics department and has hired some engineers there. The Senate recognizes the problem of lack of expertise at NHtSA and they are slowly working to solve it.

Manager

@bsnguy perhaps one of the elephants in the room is the profit margin pressures in the auto industry.  Might be a source of institutional pressure to cut corners.

Rookie

I think it's important for NHITSA to have something similar to FAA

Blogger

I wonder what the SAE (Soc of Auto. Engs.) is doing to foster/standardize SW quality.

Manager

Well NHTSA has provided the 1989 NHTSA sudden acceleration report and the redaction of the NASA report and Secretary of State La Hood has exonerated Toyota's software and the car companies take shelter behind NHTSA so to that extent they do need to be called to account

good idea, Susan.  since cars have become computers on wheels, the IEEE should be involved.

Blogger

>>As far as I can tell, the problem is more or  less the same in other companies.

This could suggest a lot of it is human error after all...

Blogger

For the engineers out there (I am one) we need to understand the underlying issues that can cause failures and work to reduce them to reasonable limits. This is what the aerospace industry does; it's what medical device companies do... it's not new teritory... except maybe in automotive electronics/systems.

Rookie

I think all sorts of upcominf ADAS could turn into a litiginous nightmare...

Manager

>Let's blame the auto industry, the component manuafactures and the engineers who don't do the proper analysis.

Systems used to be tested to be safe in all possible scenarios, but as they got more complex, engineers have fallen all the way back to just getting it to work one way, then doing testing until its time to deliver.

Blogger

JRMHess, exactly. that's why they handed things over to NASA

Blogger

I take it the kitchen sink approach is no longer something Toyota does. Obviously standards are happening organically in the auto industry...but I think embedded systems software experts should work with automotive engineers from all companies and come up with recommendations. Maybe it should happen through IEEE.

Blogger

The NHTSA does not have enough embedded software engineers - so a funding problem as well

Rookie

There's a NASA report about a Toyota throttle malfunction due to whiskers in the pedal assy

Manager

how many more issues like this will come up when auto-parking, collision avoidance, autonomous driving, etc. are more prevalent? Very complex software sitting on top of very complex hardware all being used in a harsh environment.

 

Rookie

The cultural issues explain the behavior but do not excuse the behavior. When lives are at stake, one cannot hide behind  cultural discomfort.

Manager

And looking at the hardware side of things, there's the tin whisker issue just for starters.

Manager

As far as I can tell, the problem is more or  less the same in other companies.

@Garcia-Lasheras one bad pointer increment and your program is dead or worse, misperforming.  No stack checking.  Manual memory management. Poor type checking.

Not the venue to discuss software language short comings though I think.

Rookie

but the cultural difference should not absolve them for not coming forward -- if indeed they felt that they might have made mistakes

Blogger

This is not a NHTSA problem. Assume for a moment that all of our cars will have more electronics in them next year than today... will NHTSA ban them all from our roads? The problem is what's been said, before... there has not been adequate attention to safety and reliability analysis on the systems in the car. Let's not blame NHTSA. Let's blame the auto industry, the component manuafactures and the engineers who don't do the proper analysis.

 

Rookie

when will Barr's 800 page report be published?

Rookie

I heartily agree that NHTSA needs a big push, but huge efforts by individual citizens to push NHTSA will not work. Sorry.

Instead, Congress must be pushed hard, and they will push NHTSA.

Manager

how has the automotive trade press been covering this issue?

Blogger

I agree with Antony. I think the report needs to be released un-redacted.

Blogger

Junko, the documents are written by people inside Toyota. They wrote their own cultural narrative.

Manager

>>I wonder if everyone agrees here that we want NHITSA to step up their efforts -- to look into this

To look into what exactly?

Blogger

just a thought, its likely that some of the software and systems come from Germany, India, US, Brazil, etc.

Rookie

@dabeberman: "the fragility that both C and C+ have"

Are you talking about being very prone to memory leaks? Which other flaws do you think C/C have?

Blogger

Yes of course NHTSA should be held to account for the way in which they agreed to redacting the 2011 NASA Report.

maybe the NHTSA has an ombudsman to whom this chat could be sen,t requesting a reply from them as to their plan of action going forward.

Blogger

Interesting. Difficulties on a basic language level, or more of unspoken assumptions & conventions?

Manager

I would loathe to see anyone pinning any blame on the cultural issues here, though 

Blogger

Yes, Michael, I do believe Japanese cultural issues have been at play all along. Quite a number of the documents put across an "us vs. them" attitude towards Americans, for one. A kind of samurai view of things.

 

Manager

my wife drives a Nissan Pathfinder, my daughter a Jeep, I rent Toyota Camry's.  Would hate to lose someone to a correctable software bug, as did happen, and could happen to any loved one.  Doesn't that speak for itself regarding getting NHTSA involved?

Rookie

Talk about languages--with a global supply chain in the auto industry, I sometimes wonder about the human language translation of the documentation of all the software and hardware. Certainly that is a tower of babel that could lead to many mismatches between hardware and software components. Among Toyota's internal documents that I have seen, one introspective one by a senior quality executive mentioned the terrible communication difficulties with overseas suppliers.

Manager

Betsy, do you feel there are Japanese cultural issues at play here?

Manager

In my humble opinion, NHTSA totally dropped the ball

Blogger

I wonder if everyone agrees here that we want NHITSA to step up their efforts -- to look into this

 

Blogger

@Michael Dunn perhaps in terms of "separation of concerns", but not in terms of safer languages.

Rookie

JRMHess, Michigan case is not part of the MDL, correct?

Blogger

quick comment on languages, there have been more than 100 languages since C was invented. Many of them remove a lot of the fragility that both C and C+ have.

Rookie

I'm still learning about AUTOSAR. Would this standard help?

Manager

Re. Toyota - what's happening next then? In the February Michigan case, will there be any restrictions on Barr's testimony as we'll have them in the St. Ana case?

Rookie

I have written a report on a Mitsubishi Sudden Acceleration incident in New Zealand which I would be happy to share with anyone who contacts me.

antony.anderson@onyxnet.co.uk


Also look at my website on my sudden acceleration pages where I have gathered a lot of information over many years:


www.antony-anderson@onyxnet.co.uk

@MeasurementBlues Can't wait for the self driving car bugs to show up.

Some say self-driving cars will actually be safer by removing "driver error"--the most common cause of accidents--however, for my dollar I want a "death proof" cage around my seat in a self-driving car!

 
Blogger

>>Other manufacturers also have sudden acceleration problems and it would seem that there are certain systemic elements that may be to some extent independent of the quality of the software.


So Toyota is not necessarily to be singled out here for having products of "unreasonable quality?"  If not, on what standard is
"unreasonable quality" based?

Blogger

What comes to my mind with the question of software for both Toyota and self-driving cars is both architecture/design and understanding safety/failure modes.  Although the details are vague, it sounds like there were both architecture/design issues and safety issues.

Rookie

 

Which language would you use then??

 

I'm not the expert. But I know enough to know what NOT to use :-}

Manager

Provable languages. Model-based design. And just best-practices SW engineering.

"SW engineering" – an oxymoron in most cases. But that's what we need.

Manager

JRMHess, as to how Toyota compares to others,  I don't think we really know... and that may be an opening for the future "peer review" requirement for the auto industry...

Blogger

@Michael Dunn: "I wouldn't use C for safety-critical code"

Which language would you use then??

Blogger

>Can't wait for the self driving car bugs to show up. I for one won't be trying them out in the first 5 years or so...

I'll wait 10 years, thank you very much.

bsnguy there are lots of things that can fail in the hardware

You are right. I got rear-ended by a car with "sudden acceleration" but it was a restored hot-rod under total manual control, hense no software to blame.

Blogger

The gas pedal on my 1999 Camry used to stick, requiring a tap to get it unstuck. Mechanic said they add did that, nothing could be done. It stopped by itself a few years ago.

autnomous cars....that's been the focus of our forum discussions, related to software bugs in Toyota

Blogger

@Michael Dunn that's been our position for the last decade

Rookie

Can't wait for the self driving car bugs to show up. I for one won't be trying them out in the first 5 years or so... And google will be checking the code very carefully too. Maybe a search engine for code bugs is their next big project...

Blogger

So how does Toyota's software and practices compare to that of other automakers?
We are seeing more and more problems here - Nissan had a software related recall, and so did Honda, Hyundia/Kia, GM and Chrysler - all this year!

Rookie

MISRA SHMISRA! I wouldn't use C for safety-critical code, period.

Manager

Cars are not fighter jets or commercial airliners.  Has there been the same type of focus on both software and system safety in the automotive space?

 

Rookie

Wow that is suprising about them not following MISRA!  (There seems to be some lag in my refresh)

Blogger

I think there are lots of things that can fail in the hardware: electrical noise (standard stuff), radiation, metastability, etc. It's not just a bad speed signal... it's a valid speed signal but the software reading inconsistent states based on these errors, or having values change when they shouldn't, etc.

 

Rookie

Other manufacturers also have sudden acceleration problems and it would seem that there are certain systemic elements that may be to some extent independent of the quality of the software.

My next car choice in quickly being winnowed down.

Manager

@embeddedbarr -- that's very surprising to me, given some other information I have.

Rookie

There are some VERY nasty looking Hyundai behaviours out there - search on YouTube!!!

Manager

There are software standards for automotive such as MISRA that many auto manufacturers adhere to

Blogger

in Toyota's case, there were some cascading issues... that should be taken into consideration

Blogger

@LarryM99, so we get our sensors changed when we change the oil?

Blogger

Toyota apparently used about 5% of MISRA!

Manager

MISRA coding standards are good for what they are.  Maybe they are the body to promote new standards for the automotive industry?

 

Rookie
@rich.pell t's hard to prove a specific failure mode/case
 
Maybe the "black box" recorder should be made to record more data for use afterwards when tragedy occurs?

 

Blogger

What nobody seems to have done is to look at the throttle control system from a control systems point of view. How stable is it? This could be done in the normal way that you might assess the stability of a system. This perhaps is something that could be discussed later.

So how does Toyota's software and practices compare to that of other automakers?

Blogger

You also have the situation where systems degrade over time. Sensor connections get noisier.

CEO

So MISRA is not enough.

Blogger

@Susan Rambo there are some software standards from MISRA.  Toyota did use these. However they are not formal methods based and wouldn't pick up things like unprotected variables that I believe Michael Barr wrote about

 

Rookie

>

but what about providing a degree of probablilty?

 

That's kinda what an FMEA does for HW, and what Barr investigated for SW. Not sure how you'd convert his findings to a definite probability, but I wouldn't want that SW in a word processor, let alone a car!

Manager

ISO 26262 is the auto industry functional safety standard. There's a dim possibility of parts of that being turned into regulations or referenced in regulations.

 

Manager

Quite right not to focus just on the software. For example, electrical intermittencies can cause false speed signals and it sounds as if a false speed signal might under some circumstances trigger a task death.

To get regulations, you need Congress to be willing to regulate the industry. Who can make Congress regulate automakers? Any volunteers here?

 

Manager

Good question LarryM99 - formal methods static analysis tools are used in some development processes much more than others

 

Rookie

Regardless, I think the auto industry should have basic sw standards -- regulations about software quality. Exactly...I agree with @dabeberman

Blogger

It's hard to prove a specific failure mode/case, but what about providing a degree of probablilty?

Blogger

Somebody knows if Toyota's control unit includes hardware redundancy?? -- something like "three module redundacy"

Blogger

One of the articles I read (sorry, can't recall which) said that in all the Toyota court cases where SW was blamed by the plaintiff, Toyota settled! Hmm...

Manager

Software failed, in this case, because the hardware under it failed in some way. Don't we need to fix both? Software cannot overcome all hardware failures. What where the underlying hardware failures? Were there any details? I have some guesses...

Rookie

@dabeberman FAA and similar agencies in other countries.  A large body of work exists on how to develop and test such software

Good point, since aircraft going to drive-by-wire had to give up manual overrides, which seems relevant here too.

 

Blogger

does anyone know whether Toyota changed its source code after the recall in 2010? Or improved its system?

Rookie

Lol ... the software engineers always know that the point of failure started with hardware ;)

Blogger

Is this going to be enough of a trigger to rekindle interest in provably correct software?

CEO

@Antony Anderson please send the article to me at dbeberman@aicas.com.  Thanks

Rookie

@rich.pell, hardware engineers know that the problem is alwasy software

The problem is that the auto industry is nowhere near as advanced as other industries insofar as standards of electronic functional safety is concerned.

This is perhaps an issue that could be picked up later. I have sent Junko an article written by Keith Armstrong, Brian Kirk and myself on the subject which I could perhaps be circulated to anybody interested

But in something so complex, it's hard to prove a specific failure mode/case.

Manager

It wasn't proven  - the testimony only said that there are problems with the software whihc could have caused ua

Rookie

Jacob and Rich... YES! I understand that underlying issues (that I'm unsure were adequately discussed/investigated) caused the software to fail. That is, hardware failures (bit-flips) caused the software to fail. Yes, the software could/should be improved, but what about the underlying issues that "strange things happen" that we need to understand? All of this focus on the software is fine, but I posit it's impossible to write software that can account for every possible random bit error the system might encounter (some of which are permenent, some are transient). Don't we need to understand the fundamental issues of what caused the software to fail in the first place to even begin to understand how to minimize the risks associated with those failures?

Rookie

What is "significant?"

Blogger

@Michael Dunn

Right, whenever software in control its virtually impossible to test all situations, thus we are doing their field testing behind the wheel!
Blogger

Don't know how relevant this is. We work on the standards for safety-critical software adopted by the FAA and similar agencies in other countries.  A large body of work exists on how to develop and test such software.

 

Rookie

in the software analysis done by Barr

 

Blogger

I don't think there's any doubt that the SW has a significant failure potential.

Manager

In 2010, Toyota engineers admitted to Congress that they could not test the software to be certain SUA could not happen.

Manager

Someone commented that it is shoddy software engineering ..... is it bad engineering or have these systems become so complex that validating the interactions of these software and hardware systems is pushing the boundries of the processes and test methods that have been traditionally used?

 

Blogger

Where was it proven that the software flaws were definitely the problem?

Blogger

MD, that is true...we are all dealing with probabilities, but that does not save people's lives

Blogger

You can test software (&HW) for thousands of hours, but that doesn't prove much when it'll be used for billions.

Manager

What about the overall statistics?

Blogger

Senator Grassley (R-Iowa) is one person in a position to holler about NHTSA. The Senate Commerce Committee could also do it.

IMHO, they will do nothing unless there is a strong public outcry.

 

 

Manager

Are there any doubts still lingering in any engineers' minds that the software flaws were the problem? I think the testimony was pretty solid.

Blogger

I chuckle when someone says "well, I've had my "carX" for 10 years, and nothing bad has ever happened. Minimal grasp of statistics...

Manager

Antony, yse, that's correct

Blogger

I think what NHTSA can be called to book on is their "exoneration" of Toyota in Feb 2011 when the NASA report was published.

antony.anderson@onynxet.co.uk

 

"supposed to" is a squishy concept.

 

Manager

well, that would be ideal, but unless there is oversight, sometimes the cost gets in the way

 

 

Blogger

I think ASIL is just part of a bigger standard (26262?)

Manager

Many of the automotive electronics standards are universal, I think

Blogger

MD, there could be shoddy sw in your car, but the right conditions to expose it have not arisen.

 

well, the assumption is that there are standards in place, and responsible automakers and parts suppliers are supposed to comply with them. Correct?

Blogger

Hi...Does Europe have better regulations the US on auto safety-critical software?

Blogger

there are things like ASIL, right?

Blogger

Michael Dunn, regarding safety-critical software, could you connect with me after this chat? dbeberman@aicas.com

 

Rookie

that's a good question -- who is supposed to check that?

Blogger

There are many standards for auto & safety-critical work. I don't know how compliance is checked though...

Manager

I don't think anyone at the NHTSA promised anything at this point

Blogger

I (and many others I think) would like to know how widespread this kind of shoddy software engineering is...

Manager

If anyone can share what are the most important things we have learned from this Toyota case, I want you to share them with us

Blogger

So there is a possibility that the NHTSA might investigate, maybe next year?

Rookie

Let's step back; I actually want to start from the top

Blogger

OK. we want to dive into NHTSA?

Blogger

What we've learned from the Toyota case

How we can mitigate the problems


where we go from here 

Blogger

To JRM Hess:

Great questions about NHTSA's response to Barr's findings.  I've been to DC twice to push this issue, and have observed and learned a lot about NHTSA (although still not an expert—the Center for Auto Safety has the world's best expertise about NHTSA).

First off, NHTSA responses are normally measured in months or years, not days or weeks. Second, as a policy, NHTSA does not accept materials from outside the agency unless they have a "docket" open on a particular subject, to which the public can contribute materials. They were recently asked to open a docket on SUA and they refused.

I think they will re-do their Toyota investigation only by order of Congress, and so far, Congress has not reacted in any way, AFAIK. The best thing you can do is to contact your own representative and Senators and press them to propose or support a new investigation. The main committees in charge of NHTSA oversight are the Energy & Commerce Committee in the House and the Commerce Committee in the Senate.

Manager

I would like to begin our chat today on the followings

Blogger

I have invited 

Antony Anderson who has been working as an independent electrical consultant specializing in electrical machine and control system failure investigations and expert witness work. He has been very active in our forum where our readers have been discussing Toyota cases recently.

 I also invited

Betsy Benjaminson, who worked as an official Japanese-to-English translator at Toyota; who has now become a public safety advocate

Blogger

 

why too soon - the evidence is here?

Rookie

Junko, I am here.

Betsy

Manager

Oh, here you are Antony. 

Blogger

I don't see them here yet, but I have invited a few guests today here...

Blogger

Far too soon for NHTSA to act

First, I would like to thank you everyone who joined here today

Blogger

OK, are we all here now?

 

Blogger

JRMHess, that's a damn good question

 

Blogger

My quicktime keeps crashing on Firefox.

Rookie

If Barr's finding show that there is a problem with the electronic throttle control, why has the NHTSA not acted on this? Are they planning anoterh investigation, this time with Barr's evidence?

 

Rookie

@mesamek275

The cross-examination of the expert witness by Toyota was really weak.

This could  indicate that Dr Barr was proving a very credible witness under cross examination. Pressing matters further on the points that you suggest might have been counter-productive and reinforced Dr Barr's credibility with the Jury rather than undermining it, as the defence might hope. So they would move on and attack from a slightly different direction. 

 

The cross-examination of the expert witness by Toyota was really weak. They did not contest the main points of the deposition, that is possibility of stack overflow, race conditions among some of the 1000+ global variables, "kitchen-sink" design of the 1-milliseond task, etc. I am really interested to hear why Toyota allowed such unprofessional software and what they are doing now to improve their software development practice. 

Rookie

I just got a confirmation from several knowledgable people on the Toyota case will be joining us. Including Antony Anderson from U.K.

Blogger

Hi Crusty 1 and Bert_home:  I hope you cna join us for the live chat tomorrow on Toyota unintended acceleration cases. Junko Yoshida and Michael Dunn (who wrote EDN's in-depth story) will be there to discuss.  Wed. Nov 6 -- 10amPT, 1pm ET...right here.

Blogger

@bert_home: So far my Hyundai IX35 and previously Trajet have performed without fault

CEO

I dont have a Camry but I know people who do.  I'd be very interested to hear about how common this sort of crappy design is used through out the automotive industry or if Toyota is the worst one.  Any info on Nissan, Hyundai and VW designs would be appreciated. 

Rookie

Join EE Times editors for a live chat on Toyota's unintended acceleration cases an Wednesday Nov. 6 at 10am PT/1pm ET.   Editor Junko Yoshida will discuss aspects of the Toyota Oklahoma case and what it means to both the engineer and the average consumer.

Blogger


Flash Poll
Top Comments of the Week
Like Us on Facebook
EE Times on Twitter
EE Times Twitter Feed

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)
EE Life
Frankenstein's Fix, Teardowns, Sideshows, Design Contests, Reader Content & More
Max Maxfield

Energizing the Young Engineers of Tomorrow
Max Maxfield
12 comments
It doesn't seem all that long ago when I was a bright-eyed, bushy-tailed young engineer. Now I feel like an old fool, but where are we going to find one at this time of the day (LOL)?

Jolt Judges and Andrew Binstock

Jolt Awards: The Best Books
Jolt Judges and Andrew Binstock
1 Comment
As we do every year, Dr. Dobb's recognizes the best books of the last 12 months via the Jolt Awards -- our cycle of product awards given out every two months in each of six categories. No ...

Engineering Investigations

Air Conditioner Falls From Window, Still Works
Engineering Investigations
2 comments
It's autumn in New England. The leaves are turning to red, orange, and gold, my roses are in their second bloom, and it's time to remove the air conditioner from the window. On September ...

David Blaza

The Other Tesla
David Blaza
5 comments
I find myself going to Kickstarter and Indiegogo on a regular basis these days because they have become real innovation marketplaces. As far as I'm concerned, this is where a lot of cool ...