cover story

Streamlining HDL Code Coverage Analysis The right code coverage tool and a new methodology helped designers verify nine unique ASICs thoroughly--and with fewer sleepless nights. by Steve Forde, Steve Bishop, and Ramnath S. Velu

One of the major problems facing IC designers is the verification of their designs. In particular, engineers face constant pressure to deliver larger and more complex designs in shorter design cycle times and still ensure that the devices perform the desired functions without flaws. Tools that help shorten the design time have now been available for many years, but few have aided design and verification engineers in answering a key question: How do you know when you've run enough tests? Often, engineers apply subjective measures to decide when they have done enough simulation. Certainly the designer's knowledge of the design is valuable in such an assessment, but there's still a fair amount of guesswork in deciding whether the test bench or test cases have done a good job of exercising all the corner cases of the design.

A common approach to becoming more comfortable that a design is thoroughly tested is to simply throw more tests at the design--sort of a "code and pray" approach, as in coding some more tests and praying that you've thought of all the right test cases. But that approach is often ineffective for two reasons. First, as designs become more complex, the probability of missing cases increases, regardless of how many cases an engineer can think of. Second, extra tests add to regression testing overhead, especially if they're run frequently; the overhead can add to simulation times and tie up valuable simulation licenses and computing resources.

Fortunately, tools now available take the guesswork out of determining when enough tests have been run. HDL verification coverage--or more commonly, code coverage--provides a quantitative way to objectively measure how well a set of tests covers the designer's HDL code. Code coverage tools eliminate the guesswork and inefficiencies of the "code and pray" approach.

At Intrinsix, we used DAI Coverscan, a code coverage tool from Design Acceleration, Inc. in San Jose, for a customer's project that contained nine unique ASIC designs. The tool provided more efficient and complete code coverage than traditional methods could have. For future designs, in fact, we plan to integrate automated code coverage earlier and more thoroughly into our design process, thereby reducing the uncertainty in our verification environment.

The goal of the project was to deliver Verilog descriptions and a set of functional test patterns that validated the functions for each of the nine unique ASICs on the project. A team consisting of engineers from Intrinsix and the customers designed and verified the ASICs.

The ASICs ranged in size from 50,000 to 250,000 gates. A single engineer developed the simpler chips, performing both design and verification tasks. For the more complex ASICs, two or more engineers performed the design and verification tasks separately. The overall design effort, from specification to completed code and test patterns, lasted approximately nine months.

The ASIC development methodology used in the project is typical of projects involving several ASICs (see Figure 1). We developed a set of test patterns that we handed off for use in the physical design and verification.

We had to ensure that the test patterns completely exercised the Verilog code for each ASIC.

Well into the project it became apparent that we needed a way to quantitatively determine the effectiveness of the test patterns. Specifically, we wanted to answer two questions:

• Did our test plans cover the functions as specified in the ASIC specifications?
• Did our test cases (developed from the test plans) adequately cover the HDL code of the ASICs?

The obvious answer would involve HDL coverage analysis, which we added to the methodology relatively late in the project development cycle. We decided to have a single engineer perform the HDL coverage analysis for the entire project, rather than force each design or verification engineer to learn the tool. That decision minimized disruptions, allowing the engineers to focus on their specific tasks.

Figure 1	ASIC design methodology
The basic design flow used for the nine-ASIC project is typical multiple-ASIC design. Code coverage was added to quantitatively determine the effectiveness of the test patterns, which were used in the physical design and its validation.

Working alone

We decided that the coverage analysis engineer wouldn't be allowed to modify the HDL code. To eliminate bias, the engineer neither worked alongside the design team nor was told much about the functions of the ASIC. The isolation thus served as another independent verification vehicle, as well as a means to monitor design completion.

These decisions determined how we were able to use the coverage tool. First, because we had a single engineer working on nine different ASICs over a relatively short period, we opted to try to automate as much of the work as possible, using scripts and batch-mode control files. That meant that we took little advantage of Coverscan's GUI interface, including the run set manager.

Second, we made extensive use of analysis control files rather than metacomments, which can simplify the analysis process by allowing the user to provide Coverscan directives within the source code. Since the designers weren't familiar with the tool, control files offered the more efficient option. The coverage analysis engineer did occasionally have to perform some iterative work to adjust the control files, though, such as excluding a particular piece of HDL code from the coverage run.

Finally, we had to choose between serial and parallel merging of multiple simulation runs. Serial merge allows the user to preload the results of an earlier run before starting a new simulation. Parallel merge allows the user to combine coverage statistics from multiple runs after simulation, enabling the designer to focus on a specific part of the design and then combine the results with other runs to assess the overall coverage. We opted for parallel merge because the design and verification engineers, in an effort to improve coverage, often changed existing tests (sometimes several at a time) rather than adding new or incremental tests. We also made some changes to the test pattern files, not to improve coverage but to respond to bug fixes. We found it easier to simply run the entire suite of tests again, rather than try to pick and choose the specific tests that had been modified.

Running for coverage

HDL coverage analysis tools can be used in several different ways [see "HDL Verification Coverage," June, p. 46]:

• to determine when enough simulation has been performed
• to create an efficient regression test suite--one that provides maximum HDL coverage with as few tests as possible
• to determine if more test cases are needed to exercise the HDL of the device under test

For the current project, our primary goal was to satisfy the last condition--namely, to determine if our test benches and cases provided adequate coverage for the ASIC Verilog code.

Running the Coverscan tool involves three main steps (see Figure 2). First, Precoverscan instruments the Verilog HDL source code by altering it to facilitate collecting and recording the coverage data. Next, the simulator--using the instrumented HDL code along with the test cases devised to exercise the source HDL--simulates the test patterns. Finally, Coverscan generates reports from the simulation results.

Figure 2	Coverage analysis methodology
Coverage analysis consists of three basic steps, which involve generating instrumented Verilog source code and control and argument files; capturing coverage information for each test case; and processing the coverage data files and producing reports, assisted by user-provided control files.

We automated the entire process and ran it in batch mode, which we found more efficient than the interactive GUI once we had a couple of runs under our belts. Combining our scripts into a single file (see the Listing) allowed the coverage jobs to run with minimal or no user intervention. The top-level do_everything shell script automates the Coverscan steps. Replicating the .rct file with a different run name for each of the test cases could generate a control script that supports sufficient automation and reuse. The sed's command shifts in a new name for the test run name field of the .rct file when the test is executed.

Precoverscan instruments Verilog source files for simulation and coverage analysis, creating an instrumented Verilog source code file (.pcv) out of the Verilog source files. The file contains the original source code and Coverscan commands that tell the program how it should record various events during simulation. Users can insert the commands in three ways: through Coverscan system task enables, Precoverscan command line options, and optional recording control files. We chose the command line options and recording control files because they don't require the coverage engineer to make any direct modifications to the source Verilog code.

Triple coverage

Coverscan provides statement execution, decision, and expression coverage. Of the three types, the real strength of the tool lies in its decision and expression coverage, which served us well when it came to defining new test cases.

Figure 3

Detailed decision coverage report

In the example report shown, the first run failed to generate the type_reg and o_sel values required to cover lines 88­92. After the addition of a new test bench, a second run showed coverage of all but line 92--perhaps an appropriate place to use the Exclude category. A large count ratio like the one between lines 87 and 88 may in some cases (though not in this case) signal a need to revisit the code in question. (The Verilog source code has been added here for clarity.)

Statement execution coverage is a simple measure that indicates whether a line of HDL was executed or not. The tool lets the coverage analysis engineer specify which lines of Verilog are to be executed (providing a way to exclude lines of code that are not of interest to analysis). It also allows the engineer to analyze coverage based on module types or instantiations; module coverage is then determined as the sum of all the coverage results for the module for all of its instantiations. The coverage analysis might be useful for modules that will be used many times, perhaps where each instantiation uses only a part of the module's function. We opted for measuring coverage based on module instantiation, fully exercising each instantiation. Coverscan also creates statement coverage reports that can be organized by the percentage of lines covered or by the number of lines not covered. We found both coverage reporting methods helpful, since the percentage approach is not always a true indicator of the work needed to improve the coverage.

Decision coverage, also called case coverage, measures the coverage of expressions and case statements (such as if-then-else statements) that control the flow of HDL execution. Coverscan's capabilities here are excellent, handling both implicit branches within if-then statements (implied else) as well as handling case statements. Decision coverage was key in identifying many "corner" cases (see Figure 3).

Expression coverage assesses the coverage of combinatorial expressions. Breaking a complex, multilevel expression into its components and determining which combinations have been covered is important in making sure that all the corner cases have been checked. The tool evaluated and reported expression coverage in an easily understood manner.

It would be easy for an engineer to produce piles of data with a coverage analyzer, only to have to spend hours sorting through the results. By merging results and capturing both summary and detailed reports, Coverscan made it easy to identify trouble spots. The trouble spots, along with detailed reports, could then be passed on to the design and verification engineers, who could either modify or augment the test sets to improve overall coverage. For each ASIC we normally generated six reports: a detailed report (by module instance) of statement, decision, and expression coverage, and
a summary report for each. The final stage of the code coverage process depends on a user-generated analysis control file (an example appears in the on-line version, at www.isdmag.com/Editorial/December98.html), which governs the tool's reporting functions, allowing the user to determine exclusions, categories (module or instance), types of reports (summary or detail), and sort orders.

Initial coverage results for the nine ASICs ranged from 55 to 97 percent. The large variance depended greatly on how far into the verification effort the ASIC had gone before the initial coverage run. Because we instituted coverage analysis late in the design cycle, we decided that 95 percent coverage was sufficient to deliver the designs for physical design and physical verification. We averaged about three iterations per ASIC to increase the HDL code coverage to 95 percent or greater. Most of the effort focused on decision and expression coverage. Each ASIC required an average of 60 hours of coverage analysis work.

# do_everything 


#	This script invokes two other scripts:


# 	do_precover


#	do_mbuild


# Precoverscan run only once


do_precover


# Each of the following upon completion would form a specific .cov file.


#


echo Processing pci_alarm_reg_toggle ....


cp ../PATTERNS/pci_alarm_reg_toggle.tc.tsp tssi.vectors


sed 's/TEST_NAME/swalloc_run1/' WORK/coverscan.rct >!
WORK/TEST_NAME.rct


do_mbuild


#


echo Processing pci_alloc_wr_ram .....


cp ../PATTERNS/pci_alloc_wr_ram.tc.tsp tssi.vectors


sed 's/TEST_NAME/swalloc_run2/' WORK/coverscan.rct >! WORK/TEST_NAME.rct


do_mbuild


#


echo Processing /pci_auto_seed_mode....


cp ../PATTERNS/pci_auto_seed_mode.tc.tsp tssi.vectors


sed 's/TEST_NAME/swalloc_run3/' WORK/coverscan.rct >! WORK/TEST_NAME.rct


do_mbuild


#


echo Processing pci_basic_regs_test_ne ....


cp ../PATTERNS/pci_basic_regs_test_new.tc.tsp tssi.vectors


sed 's/TEST_NAME/swalloc_run4/' WORK/coverscan.rct >! WORK/TEST_NAME.rct


do_mbuild


#


echo Processing pci_force_drift.tc.tsp ....


cp ../PATTERNS/pci_force_drift.tc.tsp tssi.vectors


sed 's/TEST_NAME/swalloc_run5/' WORK/coverscan.rct >! WORK/TEST_NAME.rct


do_mbuild


#


echo Done All tsp files .


#


echo Commencing report generation......


# This will analyze and generate reports as specified
in .act file!!


coverscanb WORK/swalloc_report.act

We observed that running the instrumented Verilog code (capturing all three coverage types) doubled the simulation run times versus normal simulation runs. Since we were running the tool only a few times per ASIC, we didn't find the overhead excessive. Nor did we attempt to tune the program's performance to reduce the overhead.

Changing coverage

Largely because of the time pressures, we didn't take advantage of all of Coverscan's capabilities, which could have boosted coverage by making fuller use of categories to exclude certain pieces of code. In future designs, we'll seek nothing short of 100 percent coverage (after accounting for lines of HDL that might be excluded). To reach that goal, we'll incorporate coverage analysis directly into the design and verification methodology. Applying HDL coverage analysis will help achieve greater coverage earlier in the verification cycle. We'll use early statement coverage results to prioritize efforts to write additional test cases, concentrating on tests that exercise parts of the design not yet covered. In addition to simply measuring how much of the code has been exercised, that approach can help flush out bugs earlier in the verification process. The tool will drive the simulation effort toward covering the corner cases without causing redundant simulations.

We found Coverscan very easy to learn and use. Designating a single coverage analysis engineer worked out well on our project because of the verification engineer's remote location, but we will consider training all design and verification engineers to use the tool. The engineers who need the coverage results will thus receive them earlier in the design cycle.

Finally, though we see real benefits in using HDL code coverage to shorten the design verification cycle, we're also looking forward to higher-level verification coverage features like transaction-oriented approaches and finite state machine coverage. Transaction-oriented approaches measure whether all possible I/O transactions have been tried, especially when a transaction often occurs over several bus or clock cycles. For instance, the frequency and range of all possible SCSI bus operations or ATM cell transfers might be measured. FSM coverage helps to validate that every state has been visited and that every possible state transition has been exercised (DAI offers FSM coverage in Coverscan 3.0, released in August). Such a verification coverage tool, coupled with transaction-oriented specifications, will significantly improve the verification of complex designs.

Steve Forde is a principal consulting engineer with Intrinsix Corp., a design and consulting service headquartered in Westboro, Mass., and works as a project leader and architect for Intrinsix's clients. He worked for nine years at Digital Equipment Corp. as a designer and architect of storage products and for six years at Sanders Associates as a designer of avionics processors.

Steve Bishop has been a senior design engineer at Intrinsix for nearly a year, specializing in Verilog verification and synthesis. He previously worked at Digital Optics Corp., performing analysis and inspection of holographic diffractive lenses and directional gratings.

Ramnath Velu has been a design engineer at Intrinsix for a year and a half. He designs and verifies ASICs and FPGAs and has previously worked on ASIC verification, code coverage analysis, and software development.

To voice an opinion on this or any Integrated System Design article, please email your message to miker@isdmag.com.

integrated system design  December 1998