Design Automation

Code Coverage Analysis Works In Hardware Design

Tandem looks at software evaluation tools for hardware design verification.

by Dan Joyce

The software world has been using code coverage analysis for at least ten years to maintain the level of quality expected for software testing. At Tandem Computers Inc. (Cupertino, CA), prototype code coverage analysis software was written in PERL to allow designers to track the lines of HDL code that are exercised by the regression test suites. This software code, named Kilroy, is considered primitive compared to the very small number of today's commercially available tools. However, Kilroy was thought to be a good case study of how code coverage can be used to improve ASIC design verification.

Code coverage analysis involves running simulations on the design while a tool tracks the number of times each line of code is exercised. The ASIC design process at Tandem involves specifying the design, coding it into Verilog, then simulating the basic functions of the design in a Verilog simulation environment. Once the design has proven that it can perform basic operations, it is synthesized to gates and implemented in an emulator. The emulation team verifies the design in conjunction with the simulation team. Meanwhile, the simulation team continues to exercise the ASIC until it has passed all of the trials in the test plan. The design is then run on the entire simulation regression suite with the code coverage tool turned to the "on" position. Code coverage results are analyzed and reviewed by the designer and the simulation team. Additional tests are generated to simulate lines of code not covered by the simulation suite. Eventually, the designer signs off on the design, declaring that the simulation covers all pertinent aspects of the code. Figure 1 shows a block diagram of the design and verification process.

The experimental design We designed a 120-kgate ASIC with 20 kbits of RAM. The design was simulated principally at the system level using Verilog. We used an emulator from Quickturn Design Systems Inc. (Mountain View, CA) for verification. That part of the design flow was not used for the code coverage experiment, as the design is normally ready for tapeout when the emulator has proven that the ASIC is stable.

Initially, we did not know if the code coverage methodology would work at the ASIC level. Code coverage was already being used to track the execution of behavioral code when simulating the lower level modules (greater than 200 lines of HDL). We also were unsure as to whether it would be cost effective or even possible to exercise every line of code during simulations at the ASIC level. We found that the "unit testing" of the small modules increased the design maintenance efforts. Thus, we abandoned testing at the low level and continued testing only at the ASIC level or higher.

We employed approximately 4000 lines of Verilog code to detail the ASIC functionality. To run Kilroy, a test probe (flag) had to be manually inserted into the code. In all, 3000 probes were inserted in a purely mechanical process without consideration for the effectiveness or efficiency of insertion. The criteria for probe insertion was that the insertion must not change the functionality of the code, yet must collect all significant test data. While we felt sure that the number of test probes could have been reduced, we did not consider probe reduction important for this evaluation. In a commercial tool, the insertion process, called "instrumentation," is automatic and always occurs before simulation. The efficiency of instrumentation is heavily influenced by the different types of code coverage algorithms that are available to analyze the code.

Figure 1. The modified design flow added the code coverage loop and improved overall design accuracy.

We completed simulations at the ASIC level before the code coverage analysis tool was used. This was done intentionally so that there would be no temptation to simply "set the flags." We created a test list that we felt covered all of the functions of the ASIC described in the specification. We simulated the ASIC completely by using this test list from the top level before doing any code coverage analysis.

At this point, we used the tool to check the coverage. Surprisingly, only about 75 percent of the testflags were set. This means that after testing the ASIC for all of the functions defined, 25 percent of the HDL code was not being exercised in the regression suite.

Directed tests were then invented to exercise the rest of the HDL code. We had to look at the line of code not exercised and write a real test at the ASIC level that would exercise that line of code. We would then run the new test on the design and check to see if those lines of code were being exercised.

The last step was to make sure that the ASIC gave the correct results when exercised in this new test. It is not sufficient to simply exercise all of the functionality defined. Rather, it is equally significant that the test environment have all the checking built into it so that it may automatically verify whether or not the ASIC performed correctly through the entire regression suite.

For a sample module of code of about 500 HDL lines and 400 testflags, about 100 testflags were not set by the first regression suite. It took the designer who wrote the HDL code almost three days to set or remove 90 to 95 of those flags. The remaining testflags were removed when, on inspection, we found that the logic was not necessary. We hypothesize that a test engineer who had not written the HDL would have taken longer to write the test since he or she would have to ascend the learning curve on the code.

One manner of testing that proved highly effective in exercising lines of unused code was to slide interacting operations across each other over a duration of time. For instance, if the design can service a read and a write simultaneously, and the read and write use some of the same resources, the tests should exercise all interactions between a read and a write operation. The technique is implemented in the following manner: perform a series of read and write operations where the read is several clocks ahead of the write, the next one is one clock closer, and so on, until the write is several clocks before the read.

Additional flags were eliminated by removing logic that was either poorly written or redundant. The last five to ten flags were deemed too costly to set, due to many simulation cycles or changes needed in the simulation environment.

We found that the effort to get this ASIC from 75 to 98 percent coverage took four simulation engineers about five weeks of effort (20 man-weeks). Another week (four man-weeks) was spent reviewing the 2 percent of the logic that could not be executed with any reasonable amount of effort. This added up to 24 man-weeks of verification effort.

Figure 2. The effective number of bugs found per day decreases over time but slowly and asymptotically approaches zero bugs.

We found that once all the code was exercised in simulation, the bugs found in the emulator dropped off dramatically. Traditional ASIC simulation without code coverage analysis tools tend to have a relatively linear bug-found rate for the first period (see Figures 2). The explanation for this bug detection rate is that the initial testing of the design keeps adding new tests that are exercising previously untested code. At the beginning, all of the code is unexercised. Eventually, as more of the functionality is tested, the bug rate begins to fall and asymptotically approaches zero (see "Period 2" in Figure 2).

The bug rate was linear for the first period with code coverage analysis (see Figure 3). This was the time during which all of the functionality was being simulated for the first time. As no code coverage analysis had yet been completed, we expected the bug rates for designs with and without code coverage to be similar in the initial verification stage. Once the first-pass validation effort was complete and all the tests were written in the test plan, the code coverage tool was used to measure the amount of code exercised. The tool showed that nearly 75 percent of the code had been exercised. At this point, the validation team began writing directed tests that were aimed at exercising the unsimulated functional code. The bug rate remained constant until all of the code had been exercised (see "Period 2" in Figure 3). The code coverage tool improved the validation results, compared to the efforts without code coverage. The same number of simulation engineers using the code coverage analysis tools found a greater quantity of bugs more readily on this ASIC.

The emulator environment was our best measuring stick for design stability for this ASIC. The emulator confirmed that the ASIC was very stable at the end of Period 2, when all of the code had been exercised in the Verilog simulation environment. As the ASIC went to tape out, we continued to monitor the design with the emulator and other parts of the system hardware. The design met the specifications with no additional revisions.

When comparing this design verification using code coverage analysis with four other ASICs, the following results were found:

1. Two other ASICs used code coverage and the results were similar. The defect find rate was consistent with Figure 3, and the ASICs functioned as predicted.

2. Two other ASICs were designed without the benefit of Kilroy. These ASICs were more complex and the extra verification effort of running code coverage was considered inappropriate. The time taken to remove defects that emerged late in the verification cycle and the need to re-spin the designs caused difficulties in meeting project deadlines.

Tandem's Kilroy tool contains another feature analogous to code coverage analysis. During the writing of code, we often thought of an illegal condition that should be identified during simulations. If that condition occurred, it was flagged as an error. Examples of this are counters going too high or too low, as well as control signals running to illegal values. The designer could mark those conditions with error flags. These errors would stop the simulation when detected and alert the simulator as to what had occurred.

Figure 3. The bug detection rate stays high past the initial debug stage and suddenly drops, indicating eminent completion.

We found that this error flagging was very powerful in finding incompatibilities between modules, especially those created by different designers. The error flags were usually set when one module was defined differently than expected. Roughly 10 high-level bugs were found in this ASIC using error flags.

Another benefit from code coverage was the thought-provoking nature of the analysis. When considering how to improve the testbench following a run, it was often necessary to rethink the description of the model which inevitably led to a simpler and more understandable coding style. Consequently, testing of the model to obtain high coverage results was then far simpler.

The benefits in terms of management metrics are difficult to quantify. Having a better coding style creates an environment where the understanding, maintenance, and ongoing support of the design are under better control. This inevitably leads to a situation where considerably less engineering time will be needed in the future.

As a consequence of this experiment, Tandem has invested in a commercial code coverage product.

Dan Joyce would like to acknowledge the help and support in writing this article from James Douglas. James is the chief executive of TransEDA, a UK based company which has been carrying out research and development of code coverage in HDLs for the last 3 years. He can be reached at jdd@transeda.com.

Dan Joyce is an ASIC designer and validation engineer at Tandem Computers Inc. (Cupertino, CA). Dan can be reached at danj@mpd.tandem.com.

To voice an opinion on this or any Integrated System Design article, please e-mail your message to michael@isdmag.com.

integrated system design  January 1997