By Jack Ganssle, Embedded Systems Programming
In my last real job I developed and sold in-circuit emulators (ICEs). Talk
about a tough product! An ICE designer needs insight into every quirk of the
target processor, which never works quite as advertised. The firmware in the ICE
is huge and hideously complex, while insane timing requirements make the
hardware equally difficult; just the sort of thing that appeals to geeks like
me. But over the years, I found an even more fascinating aspect of the business:
Anyone using a debugging tool is living on the edge of product development.
They're in that difficult phase of the project where things just don't work.
Typically the schedule is in a state of collapse, the boss's temper flares, and
everyone is tired from working too much overtime. In my role as an ICE vendor, I
worked with thousands of these developers, looking metaphorically or physically
over their shoulders as they struggled to complete their projects. I found that
few"astonishingly few"followed any sort of even vaguely-disciplined process. Yet
the minority who did use reasonable methods outperformed all other developers by
astonishing margins. They delivered good code fast. The difference in
performance of the two sorts of teams was staggering.
It's incredibly satisfying to see how a little bit of information and
encouragement can change a development team. But a couple of times a month I
hear from a group who can neither attend one of my lectures nor come to an
Embedded Systems Conference because they're desperately behind on the latest
project. Further inquiry generally shows that the group is always in panic mode.
There's no time to learn how better development techniques can bring projects to
fruition on time.
Like sailors on a sinking ship, these developers are too busy bailing to fix
the leak. The water slowly rises so they bail ever more frantically. Sooner or
later they're going down, but working faster and harder staves off the
inevitable end for just a while longer.
An old cartoon shows a fierce battle, the soldiers wielding swords and
spears. The general turns away a machine-gun salesman. He complains: "I don't
have time to talk to you"can't you see we're fighting?"
The recent Boston Embedded Systems Conference, for instance, had over 75
classes on all aspects of building embedded systems. No other venue packs so
much information into a single week. About 5,000 people attended. That's a
healthy increase over last year, but, in my opinion, an appallingly low number.
We're part of a fraternity of about a quarter million embedded developers.
Why is it only 2% of us take advantage of this incredible learning opportunity?
Is everyone too busy bailing?
Why are so many firmware projects so late and so
bug-ridden? A lot of theories abound about software's complexity and other
contributing factors, but I believe the proximate cause is that coding is not
something suited to Homo sapiens. It requires a level of accuracy that is
Cavemen didn't have to bag every gazelle they hunted"just enough to keep from
starving. Farmers never expect the entire bag of seeds to sprout; a certain
wastage is implicit and accepted. Any merchant providing a service expects to
delight most, but not all, customers.
The kid who brings home straight A's (not mine, darn it) thrills his parents.
Yet we get an A for being 90% correct. Perfection isn't required. Most endeavors
in life succeed if we miss our mark by 10% or less.
Except in software. 90% correct is an utter disaster, resulting in an
unusable product. 99.9% correct means we're shipping junk. A hundred thousand
lines of code with 99.9% accuracy suggests some 100 lurking errors. That's not
good enough. Software requires near perfection, which defies the nature of
intrinsically error-prone people.
Software is also highly entropic. Anyone can write a perfect 100 line-of-code
system, but as size soars, perfection or near perfection requires
ever-increasing investments of energy. It's as if the bits are wandering cattle
trying to bust free from the corral; coding cowboys work harder and harder to
avoid strays as the size of the herd grows.
So what's the solution? Is there an answer?
opinion, getting software correct will always be a problem, and no silver bullet
will ever satisfy all stakeholders. But some well-known, though rarely
practiced, strategies offer hope.
The first is to fight featurism. A nifty feature is just a software change,
right? We can slap in a bit of extra functionality in no time at all. This
attitude has changed cell phones from basically simple devices to
million-line-of-code firmware monstrosities.
The easiest feature to implement is the one that's not there. Prune unneeded
requirements to get the code done better and faster. Unhappily, customers"or at
least marketing departments"usually demand far more than any product really
needs. Developers usually have little influence over the requirements.
Cheat complexity growth by breaking the system up into many small pieces,
each of which is as independent as possible. Minimize coupling between functions
and modules. As the firmware evolves, coupling invariably increases. We add a
few more parameters in function calls, sneak in a global or 10, and morph the
simple keypad driver into a display/keypad/LED handler. Though I struggle with a
lot of what the eXtreme Programming (XP) folks mandate, they laudably require
that crummy code gets refactored rather than beaten into submission.
Practice reuse. You won't; no one does. But reuse is, in my opinion, our only
hope of salvation. Unfortunately, reuse goes against the nature of early 21st
century capitalism, which stresses short-term profits above all. Reuse fails
unless one makes a large up-front effort in generalizing functions. Few bosses
accept the argument, "If we double development time on this project, we can
halve it on the next dozen."
importantly, use a disciplined development process.
In the olden days all products were handmade, the work of artistes rather
than manufacturers. Plows, shears, and guns were all turned out one at a time,
each was subtly different than the one the blacksmith finished just the day
before. Over time industry learned about the value of making standardized,
interchangeable parts, and ultimately about mass production. Though the retro
movement has long admired this handwork, for most products, mass production
yields far better products at much lower costs.
But not in software. We still view ourselves as artistes, cranking out
handmade code highly optimized for each specific application. While an
engineering department will never resemble a factory, the utter chaos that rules
most firmware shops is unacceptable.
I estimate that under 2% of firmware developers have any sort of process they
use religiously. It matters little what that approach may be: XP, CMM,
inspections, and standards are all valuable ways to improve the code if used
religiously. None offer much benefit when casually or intermittently employed.
Software artistes have missed the "process" boat. Most other industries use
various sorts of defined processes to work efficiently. One way to identify an
amateur organization of any sort, be they accountants, lawyers, craft shops, or
software developers, is a lack of process. By contrast, an efficient company
like McDonald's defines a rigorous way to do just about everything. Even a
teenager, using Ray Kroc's process, can crank out Big Macs that taste exactly
the same anywhere in the world.
Firmware developers must look beyond the world of software engineering for
insight into better ways to build our products. I highly recommend Michael
Gerber's book The E-Myth Revisited: Why Most Small Businesses Don't Work and
What to Do About It. Skip the book's irrelevant last half. Gerber says that
poor businesspeople work "in" the business"they're technicians who daily make
the product or service. The business can't succeed without that individual, who
may be a genius at providing a product or service but spends every day
Gerber feels that brilliant company owners work "on" the business. They build
systems, processes, and techniques so the business runs smoothly. These awesome
managers don't just solve problems, they invent solutions that eliminate the
problem forever or that automatically deal with the issue when it comes up
They stop bailing and plug the leaks.
We must adopt the same philosophy. For instance, I've observed that most
teams spend 50% of the project debugging the code. Try something else"maybe pair
programming or code inspections.
Developers often blame management for lousy development methods. They get
neither support nor direction from the boss, who is preoccupied with just
shipping the product no matter what it takes. So the developers employ heroics,
working ever harder to produce bigger and more complex products.
We can't abdicate our responsibility to do things right. Another great
management book is Quality Is Personal: A Foundation for Total Quality
Management by Harry Roberts and Bernard F. Sergesketter. The authors contend
that "working harder" is a lousy way to solve problems. The couch potato who
promises to watch less TV will probably fail. Better: make a profound change.
Throw the TV away, or put it on a timer. Dieting? Empty the fridge, buy only
low-cal foods, don't carry money so you don't eat out, but stop making futile
promises to do better.
The authors' philosophy applies to software engineering, too. Are variable
names always totally meaningless? Skip that meeting where everyone promises to
do a better job with variables on the next project; it'll fail. Instead, change
something. Maybe adopt naming conventions and software standards. Appoint a
naming czar. Build a tool that identifies stupid names.
If the boss is indeed capricious and unsupportive, if he thwarts every
attempt to improve engineering, you can still change something. Adopt a stealth
process, one that requires no buy-in from management, and one you can implement
with neither his consent nor knowledge. A great example is Watts Humphrey's
Personal Software Process (PSP). The PSP will improve your work even if everyone
else on the team is a cowboy.
the developers I talk to do have at least an inkling about how to get their
efforts on track. Some understand the potential benefits from using a reasonable
process, and others have adopted some or all of these techniques. The vast
majority, though, sheepishly admit that they don't employ them religiously.
If the pimple-faced McDonald's worker doesn't change the deep-frying oil at
prescribed intervals, all of the food will be even worse than normal. So the
company has a system that ensures oil changes happen on-schedule, all of the
time. If we don't adopt the same determination to always use a decent
process"whatever it is"our code will be a mess, guaranteed. Cranking code is, to
me, less interesting than finding better ways to crank code.
Stop bailing and plug the leaks.
Jack G. Ganssle is a lecturer and consultant on embedded development
issues. He conducts seminars on embedded systems and helps companies with their
embedded challenges. Contact him at firstname.lastname@example.org.