Breaking News
Comments
Newest First | Oldest First | Threaded View
Page 1 / 7   >   >>
asta4vista
User Rank
Rookie
Redundancy and fault-proof design
asta4vista   11/16/2013 12:29:51 PM
NO RATINGS
Seems like Toyota engineers are not aware of fault-proof design basics. Well developed in 60-s and 70-s, redundancy and fault-proof reliability is standard in high fault cost areas like avionics or nuclear station control but is almost forgotten in gadget-oriented main stream electronics. Some comments below illustrate it even more: with forgotten general principles, companies and engineers create some home-brew and "common sense" based recipies

Simon7382
User Rank
Freelancer
This is unbelievable
Simon7382   11/11/2013 6:05:39 AM
NO RATINGS
Running the break override routine on the same main processor as part of the "kitchensink" firmware is either incredibly irresponsible or shows total ignorance regarding the basics of real time software. Not even a rookie sw engineer would do this in the US. And this is the firmware of the best selling car in the US, probably one of the best selling cars of the world. It will be many many years before I would consider buying a Toyota, even though I had two of them in the past 30 years and was reasonably satisfied with both.

Wobbly
User Rank
CEO
multiple stability/security checks
Wobbly   11/7/2013 1:54:37 PM
NO RATINGS
If you go back to my original post, we always use asserts on critical data on function entry and always use asserts on returned data, and those asserts stay in the delivered code.

Asserts are fine within a single task flow, but they do not protect adjacent tasks that can be corrupted by bad behavior between asserts. Hardware protection protects against cross infection, and ECC would have helped avoid the root cause (if the root cause was a bit flip).

It comes down to having layered defenses, both for stability, but also for intrusion and modification protection.

We haven't even discussed hardware assisted stack canaries or pseudo random cache line replacement.

msamek275
User Rank
Rookie
Re: Use software assertions and leave them in the product!
msamek275   11/7/2013 12:04:19 PM
NO RATINGS
Naturally, software assertions use a different mechanism than MPUs, ECCs, WDTs, and other such hardware. But, still I think it is very beneficial to view all these mechanisms as complementary aspects of the **same** basic method.

This basic method is to intentionally introduce redundancy checks (either software-based or hardware-based) to ensure that the system operates as intended.

The problem with viewing software assertions as "another thing all together" than MPUs, ECCs, WDTs, etc. is that redundancy checks that are very easy to perform in software, but difficult in hardware, are not being done.

Too often this mindset leads to gaping security holes and sub-optimal designs. I believe that it is exactly what could have saved the day in the Toyota UA case. Please note that even if ECC was used, it would not detect memory corruption due to the alleged stack overflow or an array index out of bounds. Simple software assertions, on the other hand, would have easily detect such things.

So I repeat the main point of my original post. Software assertions are no less important than MPUs, ECCs, WDTs, etc. Unfortuantely, they are routinely under-utilized or disabled in the production code. I just hope that we could use the Toyota case to change this perception.

 

Wobbly
User Rank
CEO
Re: Use software assertions and leave them in the product!
Wobbly   11/7/2013 7:57:40 AM
NO RATINGS
Client side MPUs actually prevent resource access, read, write, or both, on chip select or address or even register level granularity. The access permission is granted based on VMID characteristcs that are driven as part of the bus cycle. The VMID characteristics are steered at the bus master by various attributes of the access, including (possibly) Task ID running on the core.

If a carved out RAM region, or a set of device registers, are reserved for a particular VMID, that is associated with a  particular TASK, then other tasks are prevented from accessing those resources even if the processor would otherwise be taking a legitimate action.

It prevents against software defects, it prevents against directed attacks on the system.

It is particularly useful in multicore systems with shared resources.

This is very different from the behavior of the CPU tied MMU.

In one of our current SOCS, which contains eight 32bit CPUs and eigth 32bit DSPs, there are roughly sixty client port MPUs to provide protection domains to the individual device and memory space that is shared between all sixteen cores.

So even if your assert becomes corrupted because of a bit flip or some other data failure that occurs outside the domain of the assert, the end point will block the access.

Each of these capabilities form a layered protection scheme. MPUs alone are not sufficient, MMUs alone are not sufficient, ASSERTS alone, are not sufficient, ECC alone is not sufficient. Together they provide a layered protection that provide defense in depth.

msamek275
User Rank
Rookie
Re: Use software assertions and leave them in the product!
msamek275   11/6/2013 5:26:52 PM
NO RATINGS
@Wobbly: I still fail to see why an MPU-detected failure is "another thing all together" than a failing software assertion. For example, an assertion might check for an array index out of bounds. Why is such a failure so fundamentally different than an attempt to de-reference a NULL pointer, which might trip the MPU?

Wobbly
User Rank
CEO
Re: Use software assertions and leave them in the product!
Wobbly   11/6/2013 1:31:55 PM
NO RATINGS
Well, there are two ECC possibilites.

1) Correctable error, which is completely allowable. That is why you use ECC, though ECC events should be tracked and thresholded. On a car, for example, ECC events that cross a threshold could trip the ECU lamp for a service error. Note, the threshold would not be a total count, but a count per unit runtime. You need to filter them over time.

2) Uncorrectable errors. On typical ECC controllers, this throws a hardware exception.

The ECC implementations that I have dealt with actually drove a bus fault on the read cycle, they could not be byassed once they were enabled.

On client side MPUs, those also throw hardware exceptions. That is why I specifically asked about client side MPUs, as apposed to traditional MMU protection at the host side.

Assertions in code are one thing, but MPUs that actually throw back a physical bus fault into the core, that is another thing all together.

As far as software assertions, we never turn them off. They are in the production code.

selinz
User Rank
CEO
brakes?
selinz   11/6/2013 1:28:50 PM
NO RATINGS
So I'm still not clear whether this task x would disable the brakes. Is that what they are saying?

msamek275
User Rank
Rookie
Use software assertions and leave them in the product!
msamek275   11/6/2013 12:24:14 PM
NO RATINGS
I am really surprised that nobody so far mentioned the use of simple software assertions.

Most people point out that ECC or MPU were not used. But these layers of protection are really nothing else than hardware-assisted assertions. I mean, what do you do when your ECC detects a parity error or your MPU detects an unauthorized memory access? Well, you execute an exception handler, which puts your system in a fail-safe state (typically a reset).

This is exactly what simple software assertions do too, except that software assertions can easily catch subtle logic errors that no hardware can detect.

So here comes my main point. Too often I see software assertions **disabled** in the production code. Interestingly, this is done by the same people, who advocate the use of ECCs or MPUs. Isn't this a bit inconsistent? How many readers of this article ship products with assertions enabled?

junko.yoshida
User Rank
Blogger
Re: software and hardware stability
junko.yoshida   11/5/2013 10:27:04 AM
NO RATINGS
@Wobbly, exactly.



I think a lot of people are surprised, too. Although the lack of EDAC in Toyota's memory devices used at that time is not the ONLY reason that led to the bit flip, it is one important factor. 

Page 1 / 7   >   >>


EE Life
Frankenstein's Fix, Teardowns, Sideshows, Design Contests, Reader Content & More
Max Maxfield

Creating a Vetinari Clock Using Antique Analog Meters
Max Maxfield
56 comments
As you may recall, the Mighty Hamster (a.k.a. Mike Field) graced my humble office with a visit a couple of weeks ago. (See All Hail the Mighty Hamster.) While he was here, Hamster noticed ...

EDN Staff

11 Summer Vacation Spots for Engineers
EDN Staff
11 comments
This collection of places from technology history, museums, and modern marvels is a roadmap for an engineering adventure that will take you around the world. Here are just a few spots ...

Glen Chenier

Engineers Solve Analog/Digital Problem, Invent Creative Expletives
Glen Chenier
11 comments
- An analog engineer and a digital engineer join forces, use their respective skills, and pull a few bunnies out of a hat to troubleshoot a system with which they are completely ...

Larry Desjardin

Engineers Should Study Finance: 5 Reasons Why
Larry Desjardin
45 comments
I'm a big proponent of engineers learning financial basics. Why? Because engineers are making decisions all the time, in multiple ways. Having a good financial understanding guides these ...

Flash Poll
Top Comments of the Week
Like Us on Facebook
EE Times on Twitter
EE Times Twitter Feed

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)