Toyota has agreed to a $1.2 billion fine to settle a US government criminal case over unexpected acceleration in Toyota and Lexus vehicles that resulted in injuries and deaths. A jury in Oklahoma found that, in one case at least, the culprit was the firmware. (The plaintiff's lead expert, Mike Barr, is giving a talk about the case at EELive!)
This payout is on top of another $1B settlement for the same problem. Bottom line: Poor firmware has cost the company staggering amounts of cash. Testimony shows that the firmware wasn’t just suffering from a bug. Instead it was reeking with problems.
One has to wonder what the engineers were thinking. Everyone in this industry faces twin pressures: to be fast and to go cheap. Deliver now, and cut engineering costs. I have no insight into how many person-hours went into the Toyota code, nor do I know the delivery schedule. But let’s look at that most recent $1.2B payout. How does that compare to the engineering effort? The NASA report talks about a code base of “more than 280,000 lines.” Mike Barr tells me there were “over a million lines of C source code.” For argument’s sake, let’s figure on a million.
The most expensive code ever written is that of the Space Shuttle, which ran about $1,000/LOC (according to 201 Principles of Software Development by Alan M. Davis). With just the most recent settlement, Toyota’s code cost them over $1,200 per line -- without accounting for any engineering effort. The difference is that the Space Shuttle’s code is the best ever written, averaging about one bug per 400KLOC, and Toyota’s has been intensely litigated. I am not suggesting that Shuttle development practices should be anyone’s goal.
Perhaps a better benchmark is avionics. It’s largely believed that no one has been killed by defective firmware in commercial aircraft, yet that code controls pretty much everything. Sure, the pilots can take over, but modern planes are fly-by-wire. The pilot flies a computer.
What does it cost to develop the fabulous software that mediates billions of passenger-miles per day in the air? Commercial avionics is done to a standard called DO-178B (supplanted recently by DO-178C). Level E applies to software that won’t impact operations in any significant way. Level A is for code that can lead to the loss of the aircraft.
How much does it cost to write code to level A? Who knows? Data is sparse and proprietary. However, most pundits figure it’s about twice the cost of typical commercial firmware. Others ("DO-178B Costs Versus Benefits" by Vance Hilderman), in this case based on data from some 150 avionics programs, claim code written to level A is 65% more expensive than that to level E. That figure includes both the engineering effort and the certification process.
This blog continues on sister site, Embedded.com.