Mars ate my spacecraft!

April 28 was last day to register for ESC Silicon Valley. Registration is now closed. You may register on site at conference. Click here for more info.

 This Embedded Systems Conference paper is from class 402: "Mars Ate My Spacecraft" taught Jack Ganssle on Tuesday, May 3rd, 2011 in San Jose, CA.

To gear heads like me the history of engineering is rich in stories and lore, of failings and successes, and of triumphs and defeats of individual engineers. I remember reading Michener's The Source in high school and being entranced by his description of how the engineers of Megiddo, near Jerusalem, dug a tunnel 210 feet long some 2,900 years ago. The city was under siege and its well was located outside the city walls. With uncanny skill, they bored under the walls, secretly, navigating with only the crudest of instruments, yet somehow targeting the narrow fount perfectly.

Even the humblest of artifacts of technology have fascinating stories. Friends still make fun of my reading Henry Petroski's 400 page book titled The Pencil. Yet even this simplest of all writing devices sports a complex and fascinating history, one of engineers and artisans optimizing materials and designs to give users an efficient writing instrument.

Then there's James Chiles' Inviting Disaster, a page-turner of engineering failures, from bridge collapses, airline crashes, offshore oil platform sinkings, to, horrifyingly, near nuke exchanges. Strangely Chiles doesn't describe the famous loss of the Tacoma Narrows Bridge, which succumbed to wind-induced torsional flutter. The bridge earned the nickname "Galloping Gertie" from its rolling, undulating behavior. Motorists crossing the 2,800-foot center span sometimes felt as though they were traveling on a giant roller coaster, watching the cars ahead disappear completely for a few moments as if they had been dropped into the trough of a large wave.


See Galloping Gertie video.

Failures can be successes. When an aircraft goes down the NTSB sends investigators to determine the cause of the accident. Changes are made to the plane's design, maintenance, or training procedures. This healthy feedback loop constantly improves the safety of air travel, to the point where now it's less dangerous to fly than walk. That's plenty strange when you consider the complexity of such a machine. 400,000 pounds of aluminum traveling at 600 knots 40,000 feet up, in air that's 60 below zero, with turbines rotating at 10,000 RPM. It's astonishing the thing works at all.

Yet the concept of applying feedback, lessons learned, is relatively new. Those behind the Tacoma Narrows Bridge certainly ignored all of the lessons of bridge-building.

Clark Eldridge, the State Highway Department's lead engineer for the project, developed the bridge's original design. But federal authorities footed 45% of the bill and required Washington State to hire an outside, and more prominent, consultant. Leon Moisseiff promised that his design would cut the bridge's estimated cost in half.

Similar structures built around the same time were expensive. At $59 million and $35 million respectively, the George Washington and Golden Gate bridges had a span similar to that of the Tacoma Narrows. Moisseiff's new design cost a bit over $6m, clearly a huge savings.

Except it fell down four months after opening day.

Moisseiff and others claimed that the wind-induced torsional flutter which led to the collapse was a new phenomenon, one never seen in civil engineering before. They seem to have forgotten the Dryburgh Abbey Bridge in Scotland which collapsed in 1818 for the same reason. Or the 1850 failure of the Basse-Chaine Bridge, a similar loss in 1854 of the Wheeling Suspension Bridge, and many others. All due to torsional flutter.

Then there was the 1939 Bronx-Whitestone Bridge, a sister design to Tacoma Narrows, which suffered the same problem but was stiffened by plate girders before a collapse.

And who designed the Bronx-Whitestone? Leon Moisseiff.

Lessons had been learned, but criminally forgotten. Today the legacy of the Tacoma Narrows failure lives on in regulations which require all federally-funded bridges to pass wind tunnel tests designed to detect torsional flutter.

In the firmware world we, too, have our share of disasters. Most were underreported, few developers understand the proximate causes and the lessons that need to be learned. The history of embedded failures shows patterns we should—must!—identify and eliminate.

Mars attacks!
Consider the Mars Polar Lander, a 1999 triple failure. The MPL's goal was to deliver a lander on Mars for half the cost of the cost of the spectacularly successful Pathfinder mission launched two years earlier. At $265 million Pathfinder itself was much cheaper than earlier planetary spacecraft.

Shortly before it began its descent, the spacecraft released twin Deep Space 2 probes which were supposed to impact the planet's surface at some 400 MPH and return sub-strata data.

MPL crashed catastrophically. Neither DS2 probe transmitted even a squeak.