Bugged? There's a write way to reliable software

Last September, Dan O'Dowd learned firsthand about computer security dangers, when spyware on his desktop captured his credit card information, leading to unauthorized charges. The founder of Green Hills Software Inc. (Santa Barbara, Calif.) is in the business of developing software — most recently on his company's Integrity secure workstation project — that's designed from the outset for reliability.

O'Dowd has strong opinions about other issues as well, including Linux and software reliability. He sat down with EE Times at the company's headquarters recently to talk about challenges facing software engineers.

EE Times: How did you get involved in software?

Dan O'Dowd: In tenth grade, I went to a National Science Foundation summer camp. They had an IBM 360 computer, and they let us submit our Fortran card decks. You punched up a card deck, submitted it and then you waited three hours to find out your syntax error. At Caltech, I specialized in compiler technology and then spent six years in industry — four of them at National Semiconductor.

EET: Those were the days of the intellectual-property fights over the VRTX and VxWorks real-time operating systems. How did Green Hills fit in?

O'Dowd: When we started, we were a compiler company, supplying compilers to the OS suppliers. Then, the introduction of Windows was a key event for us. We thought Windows would change the landscape for how a multiwindow debugger would work, with more information and a new style of graphical programming.

In 1996, we thought the RTOS [real-time operating system] business looked pretty good. The RTOS companies decided they could do their own debuggers and tools, and we decided we could do our own RTOS.

EET: Has there been a Moore's Law increase in productivity with software? Or is there a big gap between the hardware and the software?

O'Dowd: The problem is exercising all of those transistors with reliability. Moore's Law has driven the software to be bigger. The size of the software is enormous in these embedded devices, and it becomes very complicated.

Productivity has not increased at the rate of the hardware, but it has substantially increased from [the days of] writing assembly code for the 6502, which was the job I had out of college, to C++ today. That's a big improvement. Maybe it's a factor of 10, but it is not a factor of 1,000, like the hardware has done. It is in the 10 percent-a-year range.

What we've done is substituted. We used to have four hardware engineers and one junior software guy. Now, two-thirds of the development engineers are doing software. We've compensated for the fact that the software productivity wasn't there by adding a whole lot more programmers.

EET: Is that a good recipe for making reliable software?

O'Dowd: Big teams require an entirely different methodology. Today it is chaos in software, it's the Wild West for people writing code. There is no process. And everything has lots and lots of bugs in it. It turns out there is a way to write reliable software. People do it every day, at places like Boeing and Lockheed and Airbus.

Airplanes are reliable. There's millions of lines of code flying an airplane. To my knowledge, an airplane has never crashed because of a software failure. The reason is there is technology, a process, for developing to the Level A standard of the FAA [Federal Aviation Administration].

EET: Is that process widely understood and used?

O'Dowd: The software team primarily must divide and conquer. You take your software, break it up into manageable-size chunks, define and test those chunks thoroughly. You monitor those interactions so that even if one component fails, it can't bring down other components. When the program grows to a million lines, there is no fault tolerance and no practical way to debug it.

EET: Are these methodologies part of engineering education now?

O'Dowd: We can build high-reliability software. But the methodologies are not widely known. It is not taught in universities. Most companies don't employ it. It's been used in places where reliability is paramount — in top-secret communications, in nuclear weapons facilities, in aerospace.

There are methodologies of specification and testing in which you have to say what the product is supposed to do. The first thing to do is partitioning, to divide the job into manageable parts.

The critical pieces need to be partitioned from each other, so if one fails, it can be replaced by a backup component. Then you need to define what the software is supposed to do, so someone can test for it. Another group writes the test. You design your product, then test each component so it works properly.

What do the chip guys do? They write test vectors for every hardware component you can imagine!

The FAA requires that every line of code is inspected in detail. An independent team needs to review against a high level. They code-review every line.

EET: It sounds expensive.

O'Dowd: It is more expensive, absolutely, compared with the gunslinger who goes in, writes a bunch of code, compiles it and figures that it basically works.

What you have to ask is: What is the development cost per system? The next question is: Do you want quality? If the answer is "yes," then it is going to cost somewhat more than being sloppy. But you are going to get your product to market faster. And it will have fewer bugs in it. What holds products up is that they have too many bugs in them. If you do not put bugs in at all, you are going to get your product out faster.

EET: Are these techniques spreading beyond aerospace?

O'Dowd: The carmakers don't need to be convinced about reliability, because the next-generation cars will be drive-by-wire, just like the planes are now. Now cars have mechanical linkages. If they want to do drive-by-wire, the carmakers need to apply what has been learned in fly-by-wire.

Security is another key problem. We are bringing the Internet into a car at the same time that we introduce drive-by-wire. There is a bus in the car. What if a virus jumps the bus and affects the drive control software while you are going down a twisty road? There are answers to all that, but people have to learn them and use them.

EET: Outside of aerospace and automotive, what about low-cost products?

O'Dowd: There are two markets. You can build low-quality products, or you can build high-quality products and charge more money for them. I think that those commodity products are all going to be designed in India and China. They can hack out code at a lower cost than we can here.

What we can do in this country is make quality products. That is what can differentiate us. And that is why all the programming jobs won't go to India and China. If we can't do anything that they can't do, all the jobs will go there, because they cost so much less. And they will be building the low-cost knockoff products. We have to go after quality.

EET: They design high-performance chips in India. What about software?

O'Dowd: Many Indian software engineers were educated here, have now gone back to India and are teaching the state of the art. But the best software is still written here. The best software, the most reliable, is written here. They don't have the methodologies.

EET: Do people on a small budget have to go to Linux?

O'Dowd: I'm not here to knock Linux, but in my opinion, if you build it on Linux, it can't be reliable and secure.

As soon as Linux is posted, there are lots of postings about how to hack into Linux. How much does a programmer cost a company? One hundred thousand? Anybody who can afford to pay a programmer can afford to pay for the best tools from Wind River or Green Hills. People can use Linux if they want. I'm not here to slam Linux. [Yet] a lot of money, tons of money, has been spent on Linux, but it isn't being deployed. Name five profitable products that are deployed with Linux. I know you are thinking Tivo. But that's not profitable.

We're seeing a rebound. A lot of people have tried [Linux], but not much has been deployed. People tried to use it and said, "Oh, my God. It is huge! It's not real-time, so we can't meet our deadlines. And it is hard to debug it."

If you put something on a million lines of source code that is somebody else's source code, how do you debug it?

EET: Do you think all open-source software has this problem? Are you anti-open-source?

O'Dowd: I'm not anti-open-source. I don't see any reason, in principal, why that process could not be used to create secure software. But you would have to apply discipline, you would have to abide by the rules necessary to make software reliable. There is no specification for Linux. There are hardly any comments in it. There is no real, comprehensive test suite that tests everything thoroughly.

I'm just saying [that] Linux, the way it has been developed, is in fact not reliable and not secure. And if you use it in your embedded system, you will be importing a large amount of software that is not reliable and not secure, and giving people a way in.