Industrial Control DesignLine Blog
The Use Of Assertions
Jack Ganssle
11/16/2009 1:11 PM EST
Historically, though, there's been little good evidence they are effective. Till now. In a paper from Microsoft Research the authors compared the relationship between the density of the assertions and code quality.
By "quality" they meant post-release bugs, the sort of awful things customers see. It would have been interesting to see how asserts helped during development, and the effect on schedule. I'd expect earlier identification of bugs leads to faster delivery.
Figure three in the paper shows the fault density versus assertion density for each file in four products (actually two products with two releases each). These are not toy projects like the ones that pepper so many academic studies. They are real products comprising 100KLOC to over a third of a million LOC.
The result: those modules with a low number of assertions per KLOC had much higher post-release bug rates than those seeded with many assertions. The figure is a scatter plot and I don't have access to the raw data so can't do a curve fit. But by eyeballing the results it appears that systems with less than about ten to 25 asserts per KLOC experience much higher post-release bugs than those using more assertions. The benefit seems to taper off around 50 to 100 asserts per KLOC.
More analysis by the researchers concluded that assertions detected six to 14% of the post-release bugs. That's hard to reconcile with the vastly improved quality experienced by files so seeded. Perhaps, and this is merely speculation on my part, the disciplined use of assertions is indicative of better development practices overall.
Figure 4 is stunning. Frequent readers of this column know of my hopes for static analysis. Yet this chart shows assertions find at least twice the number of bugs identified by static analyzers. (Note that the analyzers are Microsoft products and may or many not be comparable to similar products offered by other companies). Currently, commercial static analyzers are frighteningly expensive. Asserts: free, more or less.
Makes you think.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at jack@ganssle.com. His website is www.ganssle.com.




Comments
krwada
11/18/2009 4:42 PM EST
The assert() macro is good, and good to use. However, where does one "assert" to? In one of my recent projects, I wrote a complete post mortem analyzer. This thing did a complete stack trace, thread dump and a snap shot of all the interrupt service states. After the analyses, the analyzer would turn off all the interrupts and do direct writes to the terminal, and if available, to the FLASH drive. In this case; the ASSERT() macro did wonders for aiding the developers. Later, we conditionally removed the ASSERT() macro as one would expect to do ... Then we found some unexplained behavior out at the customer site.
Needless to say, this product still has the ASSERT() macro enabled out in the field for all the units shipped!
However ... in most cases, (using the roll-yer-own while(1) loop) There is no good place for the ASSERT() to go.
Sign in to Reply
soundman32
11/19/2009 3:06 AM EST
I have used liberal asserts for many years and try to persuade my colleagues to do the same as much as possible, but in my current project my attitude has changed slightly.
Whereas in the past I would use assert to check parameters on function entry and other 'impossible' scenarios, my current project is the first where a crash/bug may cause loss of life.
For this project where ever I would have put assert, I now put the same check in an if but now return an error code and assert:
old:
assert(impossible==0);
new:
if(impossible==0)
{
Sign in to Reply
soundman32
11/19/2009 3:08 AM EST
I have used liberal asserts for many years and try to persuade my colleagues to do the same as much as possible, but in my current project my attitude has changed slightly.
Whereas in the past I would use assert to check parameters on function entry and other 'impossible' scenarios, my current project is the first where a crash/bug may cause loss of life.
For this project where ever I would have put assert, I now put the same check in an if but now return an error code AND assert:
old:
assert(impossible==0);
new:
if(impossible==0)
{
assert(0);
return errorCode;
}
This means that for debug builds I can break in the debugger, and for release builds, it doesn't break, but it also doesn't execute code that it shouldn't.
Sign in to Reply
Niels204
11/19/2009 4:02 AM EST
Surely it's not the asserts themselves that make the difference, but the different culture or mindset of developers who use them?
Sign in to Reply
David Brown
11/19/2009 4:38 AM EST
First off, it should be noted that the code examples used by Microsoft are a totally different kind of code than the Linux kernel mentioned, and from the sort of embedded software written by most of this website's readers. Opinions about the quality of Microsoft's programming aside, there is no reason to assume that the conclusions of this paper are valid for a wider range of software tasks and software development processes.
Having said that, asserts can often be useful. They are particularly useful during testing and debugging, and in the interfaces between code modules if they are not well specified and documented.
But asserts in general are /not/ free, especially in embedded systems. There is a big question as to where the assert errors should go, and what the software should do when an assert is triggered. In an embedded system, asserts should only be enabled during testing and debugging - if you need run-time checks on finished software, these should be at a higher level.
One thing that can be very useful, and is free, is static assertions that are evaluated entirely at compile time. Until an standard static_assert makes its way into the C and C++ standards, it is possible to get a free (though slightly developer-unfriendly) static assertion with macros:
#define STATIC_ASSERT_NAME_(line) STATIC_ASSERT_NAME2_(line)
#define STATIC_ASSERT_NAME2_(line) assertion_failed_at_line_##line
#define static_assert(claim) \
typedef struct { \
char STATIC_ASSERT_NAME_(__LINE__) [(claim) ? 1 : -1]; \
} STATIC_ASSERT_NAME_(__LINE__)
Sign in to Reply
antiquus
11/19/2009 11:13 AM EST
Like soundman32, I have migrated from ASSERT() to something more substantiative. A colleague pointed out that sending messages to the serial port of our "toaster" was not very useful as a reporting tool (if a tree falls in the forest, and no one is there to hear....). The real cost, however, was in ROM space for all the strings: by the time you get __LINE__ and __FILE__ a few hundred times, you've chewed up all your constant space, so we switched to integer error codes.
Sign in to Reply
Tony Garland
11/19/2009 12:26 PM EST
Seems like some of the follow-on comments may be mixing apples and oranges. The utilization of ASSERT() is normally for runtime check in debug builds of contractual obligations which are considered illegal in release code. By this definition, there would typically be no code to test for or handle the errors in release builds.
Part of the utility of ASSERT() is its lightweight nature. Once you move ASSERT() in the direction of catching, reporting, logging unexpected conditions in release code, then it is no longer ASSERT and has become something more sophisticated.
However, the burden of implementing the more sophisticated release error detection and logging (both in complexity and runtime overhead) is such that it partly undermines the frequency and ease with which ASSERT can be used.
We have used ASSERT extensively with good results and we have it set up so that we can enable it for release builds for our own internal testing. But ASSERT is never enabled in code that goes out to the field because such code needs to formally handle expected errors using a different mechanism.
Sign in to Reply
one_armed_bandit
11/19/2009 2:32 PM EST
1) what to do when things assert: there are two basic choices: log && return an error, or log/reboot.
I worked on a state patrol message switcher that mixed these. There was a scripting side that would report the error (ie ill-formed data) and abort the script. If the data/etc looked good, it went thru a firewall. Anything past that point was a hard reset. We met the spec of less than a minute of downtime in 30 contiguous days in a 90 day period - one reset was a few minutes.
2) I always leave the asserts in production code, unless there is a compelling reason. I did a project for a client. 3 years later, I was chatting with someone on the project. They had changed code (standard upgrade of the product) and an obscure part of my code asserted - they were able to find & fix the problem easily. Without the assert - no way. They could have tracked the bug down - my code tends to be very clean and understandable - but it would have taken a while, because the spider-monkey would have turned into King Kong by the time the bug exhibited itself. BTW - that project was mission-critical telecom equipment.
That is, in a nutshell, the benefit of asserts - catch the bug at the spider-monkey stage.
The biggest single thing asserts give you is a run-time check of your data structs. The pointers are non-null, and assert( ptr-type == expected_type ); has saved me more time than I can describe.
The other thing it gives you is the ability to check the invariants - I expect foo() to return a value 5..10 under all cases. I expect I will *never* overflow the internal buffer: assert( strlen(buf) < sizeof(buf) );
So - I suggest using a combination like we used on the message switcher - check data/format/etc of inputs, then pass them thru the firewall && do hard asserts.
Also - every struct has the first field of "UINT16 type;". if the struct is called FOO, the value in .type == TYPE_FOO, and make it a big number, like 0xDEAD or 0xBEEF - something that is unlikely to be a normal value (ie not ASCII or 0x03).
What would be interesting is to categorize what the asserts in the studied code. A glance at the study does not seem to indicate this detail.
Also, the references do not include "Writing Solid Code" by M$ press. (It hurts to recommend M$ products, and I disagree with some things in the book, but they are style issues.)
One point: if you take the asserts out for production code, your error-handling changes - how do you test the error-handling code in those cases?
Sign in to Reply
Ralf Holly
11/19/2009 3:00 PM EST
I've always been a big fan of assertions in general and Bertrand Meyer's Design by Contract in particular. To me, the reason why assertions significantly reduce the number of bugs is not the presence of the assertion statements per se; rather it is the fact that when you put in an assertion you have to carefully think about your code and any assumptions you are making about it. Using assertions positively changes your mindset.
Sign in to Reply
john_e_k
11/20/2009 1:23 AM EST
Let's understand the proposition - keeping variables within range and passing parameters correctly results in higher quality software. And it is the programmer's responsibility to place appropriate assert statements to verify these results.
Tell me again why we continue to use languages that permit these conditions to occur. Let's define away these problems by using tools that either catch them on their own or don't allow them in the first place.
Use Ada, or better yet, SPARK, and eliminate these problems. Catch the parameter passing and variable range incompatibilities at compile time and optionally include runtime range checking to catch index and range errors.
Seems like the inmates are running the asylum. And coding themselves up minivans per Wally.
Sign in to Reply
Samip Shah
11/23/2009 6:58 PM EST
Do anyone know of any just in time compiler in C which removes asserts from compiled code based on Inputs with which it is run?
if I run following kind of asserts with
assert(Input_boolean && condition);
assert(Input_boolean && another_condition);
Input_boolean "true" they are kept in the code and if set to "false" they become deadcode and "just in time" compiler can remove the deadcode at run time and run the code.
Sign in to Reply
Samip Shah
11/23/2009 7:01 PM EST
also Input_boolean value is fed as runtime input.
Sign in to Reply