"We did a few things that NASA apparently did not have time to do," Barr said. For one thing, by looking within the real-time operating system, the experts identified "unprotected critical variables." They obtained and reviewed the source code for the "sub-CPU," and they "uncovered gaps and defects in the throttle fail safes."
Further, the team ran simulations in the Green Hills Simulator. "This confirmed tasks can die without the watchdog resetting the processor." His group also independently checked worst-case stack depth. "We found many big mistakes in the Toyota analysis that NASA relied on."
The experts demonstrated that "the defects we found were linked to unintended acceleration through vehicle testing," Barr said. "We also obtained and reviewed the source code for the black box and found that it can record false information about the driver's actions in the final seconds before a crash."
It's important to note Barr Group testimony led to a billion-dollar economic-loss settlement by Toyota last December. Because of that settlement, details of the technical discoveries made back then by the experts were not made public until the Oklahoma trial. The economic-loss settlement resolved hundreds of lawsuits claiming vehicles depreciated after the company issued recalls related to faulty acceleration. Toyota still faces lawsuits claiming injury or death related to the recalls.
Task X death
Now that the experts' testimony and findings have been made public through the Oklahoma trial, let's get into details. What defects were found in Toyota's electronic throttle control systems?
Barr said that the 2005 Camry L4 source code and in-vehicle tests by the experts confirmed that some critical variables are not protected from corruption, and sources of memory corruption are present. He believes that Toyota's engineers sought to protect numerous variables against software- and hardware-cause corruptions, but they failed to mirror several key critical variables, and they made no hardware protection available against bit flips.
Stack overflow and software bugs led to memory corruption, he said. And it turns out that the crux of the issue was these memory corruptions, which acted "like ricocheting bullets."