Coverage analysis is as easy to adopt and integrate as it is essential to ensuring consistent, high-quality verification results. Code and finite-state-machine (FSM) coverage analysis tools can be installed, integrated and running in existing RTL flows in a matter of hours; new users can quickly identify unverified areas of their design and focus new test development there, speeding the verification process. And, as coverage analysis spreads throughout an organization, additional benefits will emerge, among them facilitating a uniformly qualified pool of in-house design intellectual property and providing a common frame of reference for coverage closure.
Functional coverage will play an increasingly important role when it is integrated with code and FSM coverage tools, providing a unified view of the quality of a design's verification. Users will find that over time, new and more powerful metrics such as VN-Cover's FSM path coverage will emerge to keep pace with the increasing complexity of IC designs.
Essentially, coverage analysis is based on the measurement of a design's test suite against a set of objective metrics. Designers will choose metrics based on the stage of their design, the cost of making the coverage measurement and a design group's experience with the usefulness of a particular metric. Code coverage metrics are often the first to be employed because they can be implemented at relatively low cost with automated tools and provide straightforward results.
Early in the creation of a design's unit-level tests, simple line or statement coverage will give a good overall assessment of the thoroughness of the test. Line coverage, indicating whether or not each line of HDL has been exercised during simulation, is easy to interpret and can be collected at minimal cost either with a simulator's built-in profiling capability or with an external coverage analysis tool. Line coverage is an essential first step but is not generally viewed as being a sufficient indicator of test quality or verification completeness, as many trivial functional errors can escape even 100 percent line coverage.
Later in the design process, during final unit-level, chip integration and regression tests, additional more sophisticated metrics will be added to ensure verification completeness. The VSI Alliance (http://www.vsi.org) has defined a range of code coverage metrics, including simple statement/line coverage, that are used at various stages of the design process:
- Statement coverage: How many times was each statement executed?
- Toggle coverage: Which bits of the signals in the design have toggled?
- FSM arc coverage: How many transitions of the FSM were processed?
- Visited state coverage: How many states of the FSM were entered during simulation?
- Triggering coverage: Has each process been uniquely triggered in its sensitivity list?
- Branch coverage: Which "case" or "if...else" branches were executed?
- Expression coverage: How well is a Boolean expression exercised?
- Path coverage: Which routes through sequential branching constructs were exercised?
- Signal coverage: How well were state signals or ROM addresses exercised?
Once there is enough line or statement coverage, designers will add more sophisticated control-path metrics such as branch, expression and path coverage to ensure the thoroughness of tests in covering control flow through the HDL code. Toggle coverage is often used to ensure that all block-interface signals have toggled from 0 to 1 and 1 to 0; it is also sometime used as a presynthesis guide for toggle vectors required in manufacturing test.
Leading commercial coverage analysis solutions provide most or all of these metrics in a common environment, allowing coverage closure to be assessed in a single view. Some tools even provide this single view across multiple simulators and design languages, allowing for additional flexibility in selecting design and verification resources.
With the increasing use of both internal and third party design IP, coverage analysis results have become an important independent measure of the thoroughness of an IP creator's verification process. In fact, leading IP portals such as Design & Reuse (www.design-reuse.com) now provide a way for IP vendors to publish coverage analysis results, among other things, as a measure of IP quality.
Performance is a concern for all simulator users, and much progress has been made in recent years in reducing the simulation overhead imposed by coverage measurements. Modern coverage analysis tools eliminate PLI calls during simulation run time, thus allowing the instrumented design to run at the full rated speed of the simulator. This approach results in an overhead of typically between 20 percent and 2X, depending on the design style and the metrics used. In addition, advanced coverage tools can generate instrumentation written in synthesizable RTL, allowing coverage to be collected on hardware emulators, providing for even faster throughput as well as a consistent view of verification status between simulation and emulation.
FSM coverage deserves special attention given the important role FSMs play in controlling many designs. It has been shown that developing tests to increase FSM coverage has detected difficult-to-find bugs. Traditional FSM coverage metrics include visited state coverage, which ensures that every state of an FSM has been visited, and FSM arc coverage, which ensures that every transition between FSM states has been traversed. Design teams usually aim for more than 95 percent coverage on these metrics, but it is possible to achieve even 100 percent state and arc coverage without exercising all functionality of the FSM. Thus, visited state and FSM arc and coverage do not necessarily measure the extent to which the control functionality of the FSM has been tested.
The FSM path-coverage metric overcomes these shortcomings. Although state and arc coverage are very useful measures because of their close correspondence to the behavior of the design, FSM path coverage provides a more complete representation of a design's control functionality. FSM paths are a combination of arcs that connect states in the FSM. However, there are a large number of potential paths in typical state machines, so measuring their coverage has been problematic.
A recent innovation in commercial coverage analysis tools such as TransEDA's VN-Cover is the ability to automatically extract all FSM paths and provide a concise metric that fully represents the FSM paths without explicitly listing all of the individual sequences through the FSM. Criteria to concisely represent FSM paths and their coverage:
- Separate initialization behavior from cyclic.
- Identify cyclic behavior.
- Combine small cycles into the larger cycles that may be present in an FSM.
- Allow measurement of paths reached compared with possible paths.
By identifying the cyclic behavior, FSM path coverage metrics provide a representation of the design's functionality without being lost in the details of each individual FSM sequence. Three types of paths can clearly describe FSM behavior. First is a link that is a directed set of states that does not repeat for example, an initialization sequence. Second includes a cycle that is a directed set of states that returns to its starting state without passing through any state more than once. And third is a supercycle, a longer cycle that connects with shorter repeating cycles. Supercycles are used to identify the high-level design functionality and provide a concise summary of the overall structure of the FSM. Supercycles report design functionality that is not apparent to the user when looking at the FSM.
This new FSM path coverage metric, supported by the concept links, cycles and supercycles, promises to become a valuable new measure of verification completeness necessary for coverage closure.