Look here if you want to know what Moms have to offer the embedded world. You'll also benefit from some of Jack's tips.
Momism (môm iz' em) n. 1. A brief statement of a principle passed maternally. 2. A tersely worded statement of an observation of truth: APHORISM.
The image of mom gently guiding her young ones down paths of righteousness, teaching them the basic elements of being civilized, helping with school work, is powerful indeed. Yet we still leave home ill-equipped for real life; college itself does but a poor job in preparing us
for careers and adulthood. Perhaps mom should have taught us some more lessons. Here are a few thoughts.
Keep ISRs short.
Debugging interrupt service routines is tough and, in some cases, almost impossible. Too often those expensive tools work poorly or not at all inside an ISR. Breakpoints fail because they operate at human speeds, while the interrupts come much faster. Single stepping, the old standby of many
developers, just won't work where interrupts arrive at any sort of reasonable rate. Single step in a section of code where interrupts are reenabled, and you'll likely debug different instantiations of the ISR with each step. An emulator with trace will capture the service routine's execution, but even the largest trace buffers fill quickly from loops and recursion.
A very wise friend taught me the fundamental rule of debugging ISRs: don't. Keep the routine so short, so simple, that you can debug by
inspection. A good rule of thumb is to limit ISRs to a dozen or so lines. Worst case, keep them shorter than a page. If the ISR really must do a lot of work, why not spawn a task that handles the complexity?
Non-maskable interrupt, also known as Trap, level 7, or any of a number of monikers, can't be shut off, ever. Other interrupt inputs succumb to the "disable" instruction, and generally turn off automatically when a hardware-initiated interrupt occurs. Until you explicitly turn
the interrupt back on, the unavoidable non-reentrant parts of the ISR are safe. An NMI handler, however, is never safe. Non-reentrant code will be destroyed if the interrupt recurs. Many CPUs use an edge-sensitive input for this beast, so the slightest bit of noise can create multiple false NMIs over the course of a few microseconds. And debugging tools, like emulators, often couple small bits of spurious noise into the target system. Reserve NMI for one-time events like power failure or the apocalypse.
Fill unused vectors.
Though a CPU might support hundreds of interrupt sources, each one defined by an entry in the dispatch table, we rarely use more than a handful. If you leave those unused dispatch table entries blank, any weird vectoring will crash the application horribly, leaving no trail of evidence to the root cause.
Why would spurious interrupts occur? Maybe the hardware is defective or glitchy — it's a prototype during development, isn't it? Perhaps you've misprogrammed one
of the hundreds of registers inside of today's too complex peripherals.
Better to fill all unused vectors with a pointer to a debug routine that either logs the erroneous interrupt or reaches a lurking breakpoint.
Listen — don't interrupt others.
You'll learn far more listening than talking, and the listener never puts his foot in his mouth.
Inspect rather than debug.
Bottom line: code
inspections find bugs some 20
times more efficiently than debugging
by test. Inspect the code, design, specs,
and all relevant design documents to
find problems before writing/debugging/
testing and then chucking a lot
of expensive firmware. Inspections
won’t find all of the problems, but a
well-implemented inspection process
wrings out 70% to 80% of the defects
for a fraction of the cost.
Studies indicate that in many systems 50% of the code never gets tested. It's difficult at best to devise test
conditions for every error condition/exception handler, and for IFs nested five deep. Since post-compile error rates run around 5%1 (five bugs per 100 lines of code), even a small system with 10,000 lines of code might have 250 lurking bugs after it's completely "tested." Though devising better tests is surely a good idea, inspections will bring most of these hidden problems to light.
Inspect approximately 100 to 200 lines of code per hour.
There's a sweet-spot at 150 lines of code per hour where
inspections proceed very efficiently yet unveil most of the defects.2 Monitor the inspection rate. These numbers, which come from data I've been accumulating for a number of years, suggest that inspections cost (assuming there's no benefit!) approximately $2 per line of code, or 10% of the usual $15 to $30 per line cost for most commercial firmware.
Track debugging time.
It sure is fun to crank code. But in many organizations a large portion of a project's time gets consumed in the debugging
phase. This is a certain sign of dysfunctional development. It indicates that the developers either write the code carelessly or spend too little time on specification and design.
Measure bug rates.
A few functions or modules typically exhibit most of the product's defects. We've all been there. We've all worked on a function so complex, so poorly understood, and so badly coded that we're terrified of opening it in the editor. Change a single character in a comment and the code stops
working. Barry Boehm, the software estimation guru, has shown that these error-prone functions, which typically represent a small percentage of the total code base, contain 60% to 80% of the errors. More compellingly, his data indicate these problem-functions eat up four times more effort than their well-behaved brethren. It behooves us to take data to quantitatively figure out which are bad, and then toss bad code and start again.
Face it, you're going to have problems. The
code will be far from perfect. Plan for bugs and instrument your code to find them quickly. Does your RTOS include a stack-overflow checker? Leave it enabled whilst debugging. Or seed the stack with a pattern, then stop the debugger from time to time to see if stacks are too big or too small.
Why not fill unused ROM/flash with nasty instructions, like software interrupts, that vector off to a debug routine? When code crashes it often just wanders off, perhaps into your carefully seeded ROM area.
The software interrupts and associated handler will capture the crash quickly, and in safety-critical systems can bring the system to a known harmless state.
You'd be surprised how many embedded systems, those that are "done" and shipping, access memory in bizarre ways. Writing to ROM. Reading from unused memory. Though perhaps harmless, these odd behaviors indicate lurking software problems. Consider setting up unused/extra chip selects to trigger on any errant memory access, or expand your PLD
decode logic to signal such problems by means of an extra output. Code that behaves unexpectedly is flawed, even when the symptoms seem benign.
Clean your room.
Bugs thrive in messy places.
Performance and size
Sometimes engineering costs more than faster hardware. If you're building a million of something, production costs overwhelm development costs. That's much less true for small production runs.
forget that one of the "production" costs is that of the amortized engineering. If a design decision adds a month to the project, at perhaps a cost of $20,000, then the product's price must include this additional cost: $20,000 divided by the number of units made. Does an 8051 really make sense for your low-run application? Would a bigger CPU dramatically reduce development costs? Will shoehorning bytes into an undersized code space eat weeks of expensive developers' time?
A tiny CPU is, without
question, the perfect choice for a huge range of applications. You just can't beat them for minimizing PCB real estate, recurring costs, and power consumption. And I've long been a proponent of distributing small CPUs around a board to handle small chores like I/O processing. But do understand the very real costs of working in a confined address space with perhaps under-powered tools and languages. Make CPU tradeoffs that minimize total system cost, from engineering through production.
CPU at your peril.
A 90% loaded processor doubles development time. At 95%, figure on tripling the effort. This is hardly surprising to our intuitive understanding of programming. At one point or another, we've all battled a performance-bound system by tuning every bit of code it contains, instead of the 20% that is typically responsible for most of the real-time problems. Margins minimize engineering costs by allowing us to be a little sloppy. They let us deliver a product which is not quite tuned to
perfection, avoiding the extreme costs that entails.
Create a dynamic model of code size.
Few of us really create a meaningful estimate of ROM/flash needs. Instead, we tend to ask for as much ROM as possible, or perhaps double the amount used on the previous project. It's tough to estimate binary sizes when starting a large project.
But we can't abdicate our responsibility to monitor code growth. Telling the boss a month before delivery — when the hardware design is cast in
PCBs — that we need more ROM is a sure path to career stagnation.
It's a simple matter to build a spreadsheet that lists all of the modules the system will contain with estimates of their size (in lines of code, function points, or any other reasonable measure). Edit in the real source line and object size of each module as it's completed. Over time you'll find a reasonable approximation to the number of bytes of code per line of C; have the model apply this to the as-yet-uncompleted portions of the
code to predict final system size. Odds are you'll spot ROM shortages early on, when there's still time to take design action.
Size doesn't matter.
Be content with yourself and who you are.
Reuse and maintenance
Be realistic about reuse. Reuse is hard. Good rules of thumb: Before you can develop code for reuse you must have developed it at least three times. Before you can reap the benefits of reuse you must have reused
it three times. One proposal for Reagan's version of the Star Wars missile defense system, which was pegged at 100 million lines of code, was that every module had to have been used three times before being included in the system. Not a bad idea, especially for a system so difficult to test.
Global variables are responsible for most of the evil in the world. A program infested with globals becomes non-maintainable, buggy, and a nightmare for all team members. Globals
also make reuse all but impossible.
Embedded systems suffer from another dependency problem: code that talks to hardware. Encapsulate all I/O operations.
Self documenting code does not exist. Long variable names do not self documenting code make. Judicious name selection is just a part of good coding.
Any idiot can write code. Even teenaged hackers manage to crank out working software. Professionals create beautiful code that is crystal clear and a
joy to maintain. Accurate, lucid comments are an important ingredient of well-written firmware. Code is nothing more than the computerese description of what's going on; comments are the human description.
Use active voice. Capitalize using standard English rules. Check your spelling. Describe concisely the goes-intas and goes-outtas, as well as what happens and why. Some enlightened programmers write all of the comments first, and then fill in the C at their leisure. The hard part, after
all, is creating an accurate, documented design. The code is nothing more than a simple translation of a good design into computer-lingo.
Keep compiles clean.
Don't come to the dinner table with dirty hands, and don't deliver code reeking of unpleasant warnings.
Why do we tolerate warning messages from our compiler? Firmware lives forever. When someone else opens your code five years from now for an upgrade and finds hundreds of warnings scrolling off the screen, he'll have no idea
if the messages are expected or are an effect of the way he's reinstalled the tools. Maintenance is an unavoidable aspect of the software development process; he who programs without maintenance in mind is an amateur.
Keep the code strictly ANSI compliant to minimize warnings and maximize portability. Segment unavoidable deviations from the standard to separate modules which document expected unusual compiler behaviors.
The OOP folks chant "encapsulation,
polymorphism and inheritance." Of those three, encapsulation is the easiest and most powerful tool for building well-written, easy-to-understand code. It's equally effective in assembly, C, or C++. Bind "methods" (code that accesses a device or data structure) with the data itself.
You'll miss your teeth when they're gone.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their
embedded challenges. He founded two companies specializing in embedded systems. Contact him at
1. Gilb, Tom and Dorothy Graham.
Reading, MA: Addison-Wesley, 1993.
2. Wheeler, David A., Bill Brykczynski, and Reginald N. Meeson, eds.
Software Inspection: An Industry Best Practice.
Los Alamitos: IEEE Computer Society, 1996.