design flow

Practical Approaches to Improving ASIC Verification Efficiency Code coverage analysis can simplify developing test benches as well as verify them. It can help order tests as well. by Stuart Riches and Martin Abrahams

Statistics show that software coding, even when done well, has 1 to 3 defects per 100 statements. That's too many errors for any application, let alone for ASIC design, in which undetected errors become embedded in the final product. It's not surprising, therefore, that software verification is a major aspect of ASIC development, accounting for more than half of the end product's labor costs.

One of the reasons for the high cost of testing, as E.W. Djikstra of the University of Texas, Austin, neatly put it in 1960, is that "testing can only show the presence of errors and not their absence." It's not possible to tell when the design code is free from defects, so you never really know when it's safe to stop. As a result, the tendency is to overtest, raising costs with constantly diminishing returns.

In usual practice, ASIC design and verification teams concentrate on using simulation tools to develop test benches for their designs. Their main way of doing that is to use the functional testing approach--that is, they develop test data based on documents that specify the behavior of the hardware, with little regard for its internal logical structure (see "Test Techniques"). Introducing some additional testing techniques can increase the effectiveness of that test approach, thereby reducing both the cost of and the time required for testing (see the figure).

In particular, measuring code coverage, which is now standard practice in our ASIC design handoff, can be used for more than just verification. Code coverage analysis reduces the design cycle time and provides the design engineer with fine-grained insight of simulation behavior. Once the data are close at hand, it becomes possible to make time- and labor-saving optimizations in the design cycle.

Up-front code coverage
In designing local-area network devices at Texas Instruments' Network Business Unit, TransEDA's VHDLcover code coverage measurement tool has been used so effectively that the design team proposes to make up-front code coverage analysis its standard approach. Moreover, it has plans to examine further applications for code coverage data in two nontraditional ways: first, to reduce the effort required to develop the test bench, and second, to produce an optimal ordering of tests to execute, which increases the likelihood of finding errors earlier in the process.

Accelerating test bench development
A typical test bench used at the Network Business Unit consists of a VHDL wrapper inside which the complete VHDL model for the design is instanced. The wrapper itself is programmable, its function being determined by a VHDL package body. The package body is obtained by selecting one entry from a test suite--that is, a library of test cases. Each test case in the suite is written to exercise an independent function of the design.

The process used for developing test cases relies heavily on translation of the design specification. Usually the scope of the test suite is defined at a brainstorming meeting, which produces an initial list of required tests. Individual tests have to be written and debugged, and periodically--as the design itself is constantly modified by adding new features, fixing bugs, or both--the entire test suite must be simulated to validate the status of the design VHDL; that is, it must be regression-tested (see "Test Approaches"). Code coverage analysis is normally not performed until the final regression is run, just prior to handoff.

Figuring out a priori what test cases are required is far from easy. For one thing, the initial list of tests typically covers only half the requirement. Test cases may target the right areas but lack sufficient detail to completely exercise the functionality. Worse, specification changes during the design cycle may be captured in the design itself but omitted from the test cases. The sign-off requires only statement coverage, so frequently little attention is paid to more detailed measures. Time pressures force patching of the test suite rather than structured design.

Figure	A project verification review mechanism
The regular project approach to functional testing has been augmented with structural testing using code coverage and a directed approach to code inspection.

Making code coverage analysis a regular checkpoint during the design cycle reduces some of those effects and lessens the time it takes to produce a complete test bench. During the design of the TNETE223 mixed-signal transceiver, the design team used VHDLcover to provide regular feedback to both the design engineers and to the test bench developers. The information was intended to complement what was already known about the testing requirements. The TNETE223 was chosen because a very tight design schedule required an alternative approach to ensure both functionality and completeness. Another motivation was that many of the test case writers were unfamiliar with the design, and there are few ways to encourage familiarity that are better than undertaking a code coverage analysis.

Independent testing
It's important that the test case writers haven't contributed to the writing of the design VHDL itself. Such independent testing ensures that the test cases are written to find errors and not to demonstrate that the design is wonderful. It also follows, therefore, that a third, independent group should perform code coverage analysis because it has the responsibility to find errors in both the design and the test suite.

In practice, starting from a point at which approximately half of the design VHDL had been written and at which there had been just a few test cases, the team ran coverage analysis biweekly. In recording the missing tests, it devised a simple tabular format to assist feedback of information. The table had two columns with untested functionality in the left column and tested items on the right. With subsequent regressions, the detail in the table was seen to increase. Initially, whole entities and processes were reported in the table; but as coverage improved, so did the detail, and three clear benefits of the feedback emerged.

First, they saved time developing the test bench because the code coverage analysis enabled the team to quickly identify omissions without having to fully understand the required functionality. Second, the team made use of all code coverage types--something that wasn't previously required. Doing that undoubtedly improved the quality of the tests. Third, the periodic analyses provided a useful metric toward completion and an impression of the delay that could be expected in completing the design through VHDL coding to handoff, which aided scheduling.

Test bench optimization
A full regression test suite can contain hundreds of individual test cases, so simulating all of them may require several days. While numerous solutions are available to reduce the time required--hardware accelerators, formal verification, and simulation on parallel machines, to name a few--such solutions are typically expensive and therefore not usually available to the ASIC designer. Code coverage analysis, though, may provide a solution.

The TNETE223 project revealed some interesting trends in test case development. In particular, it became apparent that test case authors tended to create new test cases when they identified small holes in coverage rather than to modify existing tests. The knowledge that the test suite contained groups of similar tests indicated that a regression might involve a large degree of redundant simulation.

Each sequence of test cases contributes incrementally to cumulative coverage. Using the TNETE223 test suite, the design team performed a test case sort, searching for the test cases that provided the most coverage. Using this ordering as the execution order in the regression run would improve the potential for finding errors earlier in the regression. Further, knowing that a certain level of coverage could be reached after a given time through the regression testing, the designers could be confident that there were no design errors, and it wouldn't be necessary to wait for regression testing to be completed before moving to handoff. This approach is a simple application of Pareto's law, which states that 80 percent of the problem can be solved using just 20 percent of the effort.

A good verification tool produces coverage results for each individual test case, segmented by a VHDL source file. For example, a coverage number (percentage) is obtained for each VHDL file for every test case that's run. If there are 50 entities in the design, one per unique VHDL text file, there should be more than 50 test cases to make up the entire test suite. A two-dimensional matrix of verification results easily shows which test cases are most effective (see Table 1).

The algorithm for ordering the tests would then be the following:

1. 1. Examine the matrix, thereby identifying tc2 as the most valuable test case; it completely covers more VHDL segments than any other.

2. 2. Extract tc2 as the first test to be executed and redraw the matrix without tc2 and also without any of the files completely covered by tc2.

3. 3. Repeat steps 1 and 2 to identify the next-most-valuable test case (tc7 in the table) and continue the iterations until the point is reached at which no more VHDL segments are completely covered by any of the remaining test cases.

4. 4. Complete the list by taking each file in turn and choosing the test that gives the best coverage for that file.

In the simple example, the design team would have identified that running tc2, tc7, and tc3, in that order, from the full set of eight possible tests, would maximize the code coverage in the early stages of a regression.

Results fit Pareto theory
With some 5,000 entries in the initial matrix, the implementation had to be automated using a Perl script. The results (see Table 2) were obtained at three checkpoints during the design cycle, and they demonstrate that a high degree of coverage could always be obtained running just a small subset of tests from the test suite.

A side effect of the method is the potential to identify wholly redundant test cases. Two tests having identical characteristics in the matrix must be the same. The TNETE223 test suite was found to contain one such test case, which had been created by copying an existing test case but never modified to verify a different function. The design team was able to remove that test from the regression suite.

A change in approach
Up-front code coverage analysis saves both time and effort. The design engineers from the TNETE223 were sufficiently pleased with the change in approach. The aggressive schedule was, for the most part, achieved, and the engineers have not had any hesitation in making up-front code coverage analysis their standard approach.

As mentioned earlier, the design team has plans to examine further applications for code coverage data. Among the possibilities they're considering are identifying test cases that exercise known critical timing paths, identifying the test cases that will produce the best toggle data for power analysis, and developing a starting set of vectors for reuse in fault-grading a design. The usefulness of code coverage analysis is just beginning to be appreciated.

Stuart Riches is an ASIC design engineer at VLSI Technology, Inc. in Milton Keynes, U.K. He was a design engineer at Texas Instruments, Inc. in Northhampton, U.K. when this article was written. Initially, he worked for four years as a software engineer developing EDA tools at TI in Bedford, U.K. In 1990, he changed jobs and worked on the 'C80 DSP team until moving into mixed-signal ASIC design in 1993, contributing to a number of local-area network devices.

Martin Abrahams is an applications support manager at TransEDA, Ltd. in Eastleigh, Hampshire, U.K. He is responsible for customer support operations worldwide. He has also worked at TransEDA for six years and assisted with the development of both VHDLcover and Coverplus.

To voice an opinion on this or any Integrated System Design article, please email your message to miker@isdmag.com.

integrated system design  July 1998