Design Tools

HDL Verification Coverage Verification coverage tools can help produce better-quality designs more quickly by ensuring that the entire design has been simulated and by eliminating redundant testing. by Dean Drako and Paul Cohen

Applying verification coverage to HDL-based hardware design is a relatively new concept that's rapidly gaining popularity. Verification coverage can help designers using HDL-based designs improve design quality and reduce time to market by providing a quantitative measure of simulation completeness.

By monitoring the execution of the HDL code during simulation, verification coverage can ensure that the entire design has been simulated. To do that, however, coverage tools must address two problems. The first is highlighting the parts of the design that haven't been simulated. The second is showing that the design has been completely verified, so that the designer can move on to other tasks.

There are a number of different types of verification coverage, each of which contributes to solving the verification problem. Generally, designers use multiple coverage techniques to obtain a complete picture of the thoroughness of the simulation.

A new concept
Software engineers have used code coverage techniques for a number of years as part of software testing. Thus those techniques are well understood. However, code coverage is relatively new to hardware design, and few publications have addressed its use with HDLs.

Several commercial tools for Verilog and VHDL code coverage are now available. The value of code coverage to the hardware community is generally higher than to the software community because of the high cost of "respinning" a chip in terms of both NRE costs and time to market. As chip complexity increases, design verification at the RT level becomes a primary bottleneck. Verification coverage analysis is used primarily during RTL simulations, where the design is typically verified for functionality. The RTL simulations are often the "golden simulations"--simulations at other levels of abstraction must agree with them. Portions of the HDL code that aren't verified at the RT level may remain unverified as the design proceeds towards fabrication.

Verification coverage can be used to measure how much simulation is needed before committing a design to synthesis or fabrication. It answers an important question that designers and verification engineers ask during simulation: Are we done yet? In a typical design environment, the verification task is completed when the engineers think they've done a thorough simulation. This typically subjective measurement of verification completeness is based on the designer's understanding of the design.

Verification coverage provides a method for designers to perform a quantitative analysis of simulation completeness before committing the design. They can determine what areas haven't yet been tested, so that they can focus their efforts on these areas. In addition, some cases may not be exercised during actual product testing, because it's difficult and costly to create the circumstances. Verification coverage can guarantee these cases are exercised during simulation. As a result, designers can obtain 100 percent coverage, increasing their confidence that their design is correct.

Furthermore, verification coverage can eliminate redundant simulations. As simulation proceeds, tests written for one purpose will often test other functions. Verification coverage information identifies these overlaps automatically.

Black box and white box testing
Black box testing (also known as functional testing) compares a program or a block's functionality with the desired functionality. In a typical hardware scenario, black box testing is the comparison of a design's actual output vectors with the desired output vectors while applying a stimulus during simulation.

Black box testing can ensure that the design functions as expected for the given input stimulus, but it doesn't ensure that that the input stimulus fully exercises the code. Using a black box strategy, the designer measures verification completeness by assessing the number of tests successfully run, the number of cycles successfully run, or the completion of certain milestones (for example, booting the operating system).

White box testing, sometimes called "structural testing," is different from black box testing, because it provides visibility into internal data and structures. Verification coverage generally consists of measurements based on several white box techniques to ensure that the code is fully exercised. However, it is not structural testing. That is, it doesn't verify correctness; it only verifies that the design is thoroughly exercised. A complete test system verifies that the design is thoroughly exercised and that it functions properly.

Figure 1	Summary screen
Verification coverage employs more than one approach. The summary window shows the application of multiple coverage types to the same design.

Designers have been applying white box techniques for a long time. Verilog or VHDL "checkers" are often inserted into a simulation to monitor internal signals or states and verify that errors haven't occurred. Designers will often enter test code into the design, using their knowledge of the design to insert the tests. All of these methods are in effect "white box" methods. Although they are useful to ensure that the simulation is operating properly, they still don't provide a quantitative measurement of simulation completeness. Using a more systematic white box strategy, verification coverage tools provide better metrics to determine the completeness of verification and to target the verification efforts.

Figure 2	Hierarchy of coverage
The verification coverage analyzer window shows the design hierarchy, categories, and coverage information. The coverage information is shown as both percentages and as the number of statements not executed.

Limitations
Code coverage itself isn't a comprehensive solution to the verification coverage problem. Code coverage, which differs from verification coverage, tracks what code has been executed during simulation. Although it's obvious that code that hasn't been executed hasn't been tested, the converse isn't true: There's no guarantee that code that has been executed has been tested. It's possible that the code was fully exercised and the results of the code execution never propagated further into the design. The results of the execution may never have been used. Imagine two computation units connected to the inputs of a mulitplexer. If both computation units are fully exercised, but the results from the second computation unit are never muxed out, it hasn't likely been verified.

Verification coverage is an extension of code coverage with improved metrics that selectively filter the raw data to increase the probability that verification has been performed. To prove that a design has been fully verified, even structurally, is difficult. However, it's possible to enhance code coverage technology significantly such that the likelihood of obtaining verification coverage is higher. Techniques focused on finite state machine (FSM) coverage, which typically verify both state visitation and state transitions, are more likely to have a correlation with verification coverage. Sometimes simple code coverage will report that a statement has been executed or "covered." Closer analysis of the expression coverage results, though, may show that the code was only executed during reset, when all of the variables were unknown ("X"). Therefore it was "covered" but not verified.

Types of Coverage
There are many different types of analysis that are typically categorized as code or verification coverage. Some definitions of "coverage" overlap, and others are completely redundant. Different names are sometimes used for similar types of coverage (see the table).

Statement execution coverage
During simulation, a recording process tracks the execution of each statement in the HDL code. Subsequently, an analysis is performed to determine which statements were executed and which weren't.

Statement execution coverage can treat conditional statements as separate statements or as a single statement. The way the statements are treated has a great impact on the usefulness of statement execution coverage. For example, the Verilog code fragment:

always @ (a or b) begin
r = 0;
if (a == 0) r = 1;
end

contains either two or three "countable" statements. The statement r = 0 is clearly one countable statement. The if statement, however, can be viewed as either one or two countable statements. If this line is treated as a single statement, it would be considered covered, even if the r = 1 code section is never executed. It's much more thorough to treat the line if (a == 0) r = 1; as two statements and report coverage of the r = 1 segment independently. In this scenario, there will be three independent statements.

Some coverage systems that provide statement execution coverage will incorrectly claim to provide decision coverage, because they've verified that all pieces of the code were executed. To truly obtain decision coverage, it's necessary to verify that the a = 0 case and the a = 1 case were both exercised. Statement execution coverage will report 100 percent coverage, even if a never equals zero. To obtain a more complete solution, statement coverage must be combined with decision coverage or expression coverage to verify that the a (does not equal) 0 case is checked.

Although statement execution coverage is usually specified as a percentage, the figure can be misleading. The number of executable statements in a Verilog module can vary widely. Therefore, when statement execution coverage is less than 100 percent, the absolute number of statements covered versus the total number of statements is also important. Consider two modules, one with one line covered out of two, or 50 percent coverage, and the other with 75 lines covered out of 100, or 75 percent coverage. Clearly, the larger module is further from completion, although the percentage is higher.

The verification coverage analyzer window shown (see Figure 2) lists both the percentage coverage for each module (the "S%" column) and the number of statements not yet covered (the "s#" column). The total number of "countable" statements in the module is also listed. (The "S%C" and "S#C" columns provide similar information for the module and its submodules.)

The source code window highlights areas that haven't been thoroughly exercised in the simulation (see Figure 3). Statements highlighted in red haven't been executed, whereas red expression coverage percentages indicate expressions that haven't been completely exercised.

Basic block coverage
Basic block coverage is similar to statement coverage, except that the measured unit of code is a sequence of nonbranching code. Most coverage systems use basic block coverage in the recording process but display the results on statements in order to simplify the user's analysis.

Decision coverage
Decision coverage measures the coverage of expressions and case statements that affect the control flow of the HDL execution--for example:

if (a == 0) begin


...


else begin


...


end

This example contains a branch that could be exercised in one of two ways. Decision coverage will report whether the if statement was evaluated as both true and false during simulation. Note that decision coverage still applies if the else statement doesn't exist, as in the following example:

if (a == 0) begin


...


end

The if statement can still be evaluated as both true and false; decision coverage will still verify that both the explicit branch (a = 0) and the implicit branch (a = 1) are executed. This is sometimes referred to as an implied else statement.

Decision coverage also applies to case statements:

case (a) 


	1: q = 7;


	2: q = 10;


	3: q = 27;


endcase

In this example, decision coverage will verify that each of the three branches of the case statement is taken. In this particular example, however, the same information could be obtained from statement execution coverage, since each leg of the case statement is a separate statement. However, in the example:

case (a) 


	1,2: q = 7;


	3: q =
10;


endcase

statement execution coverage will consider the line 1,2: q = 7; covered if a is 1 or 2 . Decision coverage analysis is needed to verify that a has been both 1 and 2 during simulation.

Figure 3	Execution count and percentage
The source code window shows the areas not fully exercised highlighted in red. The execution count (S#) indicates unexercised code, and the expression percentage (E%) indicates partially exercised structures.

Expression coverage
Expression coverage expands each expression in the HDL code and verifies its evaluation at a set of "interesting" values. It's most typical for expression coverage to examine expressions in Boolean fashion. Boolean expression coverage is the analysis of expressions broken down into basic Boolean elements. It can be thought of as examining the "decisions" that occur within the expression. For example, given a simple assign statement:

assign a = b || c ;

it's useful to examine each element and verify that its impact has been measured on the design. At a minimum, the following cases should be verified: b = 0, c = 0; b = 0, c = 1; b = 1, c = 0 . In addition, some may wish to verify the case where both b and c are 1 .

In a more complex expression, such as:

assign d = (e == 4) || (f == 7) & g;

it's important to verify that e = 4 and e (does not equal) 4, f = 7 and f (does not equal) 7 , and g has been both 0 and 1. In another complex expression:

assign a = ( b || c ) && (d || f ) && g

it's important to examine the Boolean base elements and verify that each variable in the expression has had an impact on the output. The expanded expression coverage results indicate what combinations of variables haven't been exercised (see Figure 4).

State machine coverage
State machine coverage is primarily a hardware design concept, so it's difficult to build upon software development techniques. State machine design is one of the areas most prone to the introduction of errors, because the control and complexity of state machines is generally high. Therefore designers have developed many of their own techniques to obtain and measure state machine coverage.

When using state machine coverage, designers must verify that every legal state of the state machine has been visited and that every legal transition in the state machine has been exercised. Verification can be accomplished by analyzing the changes of the state variable in conjunction with the state machine's clocking mechanism.

State machines can be very complex. Not every state (especially not every state transition) is legal or needs to be tested. During simulation, it's reasonable to expect each state to be visited, but it's unlikely that every transition between states will be covered. Manually indicating which state transitions should and shouldn't be checked can be exceedingly cumbersome. Fortunately, verification coverage tools can extract the legal state transitions automatically from the code.

Path coverage
Path coverage measures the percentage of all possible paths through the code that have been exercised. A path is a unique sequence of branches or decisions from the beginning of a code section to the end. Path coverage is similar to decision coverage; however, it handles multiple sequential decisions. In the example:

if (a) begin


	 ...


end


if (b) begin


	 ...


end

there are four paths through the code fragment that correspond to the possible combinations of a and b . It's possible to obtain complete statement execution coverage and complete decision coverage and still not have complete path coverage. Executing this sequence of code with a = 0 and b = 0 and then with a = 1 and b = 1 will obtain full statement execution coverage and decision coverage. Path coverage will report only 50 percent coverage, since the two cases in which a (does not equal) b haven't been covered.

Figure 4	Detailed analysis
The source code window shows expanded expression coverage results. Although the expression itself has been executed a number of times in the simulation, it hasn't yet been fully exercised.

Path coverage is considered by some to be too cumbersome in practice. There are a very large number of possible paths through a design, making 100 percent path coverage impractical or impossible. Consider the following code fragment:

if (a) begin


	 ...


end


if (a) begin


	 ...


end

Standard path analysis will require four paths to be exercised in this example. If a doesn't change in the code, however, only two real paths exist. Locating and justifying all of the spurious errors in path coverage data can require significant effort.

Event coverage
Most simulators are event-driven; thus it's possible for the designer to create a list of possible events that may occur in a design. (Events are typically the change of a signal.) If the events that have occurred during simulation are recorded, the designer can examine the event coverage.

A typical example of event coverage in Verilog is the always @ statement--for example:

always @ (a or b or c) begin


	 ...


end

Data can be recorded to verify that the code segment in the always @ statement has been "fired" for each of the variables it has been sensitized to. The precise usefulness of this method depends on the application. It isn't commonly used to verify sequential designs.

Toggle coverage
One of the oldest measurements of coverage in hardware design, toggle coverage measures the bits of logic that have toggled during simulation. It can be used at both the RT and gate levels. Sometimes it's used as a crude method of fault coverage on gate-level designs, and often it's used as the foundation for power analysis tools.

Toggle coverage can be used as a crude method of verification coverage as well. If a bit doesn't toggle from 0 to 1 or from 1 to 0 , it probably hasn't been adequately tested. Toggle coverage is well suited to gate-level designs. It serves little purpose to perform statement execution coverage on a gate-level representation, since there are no real "statements." At the gate level, toggle coverage provides more information regarding the thoroughness of the simulations.

Variable coverage
The extension of toggle coverage to the RTL is known as variable coverage. For a single-bit variable, variable coverage and toggle coverage are the same. Toggle coverage is typically used at the gate level, where there is no knowledge of grouping bits into a larger entity, such as a bus. For a bus, variable coverage can be used to verify that particular values have been visited. For example, it may be useful to verify that each bit of a bus has toggled, that the bus has had its minimum value (typically 0), its maximum value (all 1s), and an alternating pattern of 1s and 0s.

Dean Drako cofounded San Jose­based Design Acceleration, Inc. in 1992. From 1993 through 1995, he was vice president of business development and engineering at 3DO. In December 1995, he rejoined DAI, where he is president and CEO.

Paul B. Cohen handles application engineering and product marketing for DAI Coverscan. Before joining DAI in 1996, he was a senior design engineer at Integrated Device Technology's Atlanta Design Center, where he was also involved in EDA tool definition and support.

To voice an opinion on this or any Integrated System Design article, please email your message to miker@isdmag.com.

integrated system design  June 1998