System Design

Effective Design Verification

Testbench suites aid in the implementation of verification plans.

by Larry Saunders

Design verification means different things to different people. At SEVA Technologies Inc. (San Diego, CA), design verification is a method of determining that the performance, behavior, logical function, circuit timing, and physical characteristics of some proposed computer hardware design adhere to design intent. It is an on-going activity with a changing focus that spans the entire design schedule.

Traditionally, design verification has been considered an eccentric relation. It's not something people like to think about. While considerable thought may be put into the design of a given computer system, little thought is put into ensuring that the overall design works as intended. Such a traditional design project might proceed like this:

The project team's system architects, if there are any, tend to work separately from the logic-design group. As they architect the system, they tend to communicate among themselves using modeling languages oriented to their select fraternity. They verify their models using tests that only they understand. Given their focus on things such as algorithms and performance, the models they create will have no logic content. The architects may even refrain from modeling significant aspects of system functionality because everyone understands how it works. For these reasons and others, when it's time to start the detailed logic design, the system architects prefer to communicate design intent to the IC designers via numerous block diagrams, along with specification documents.

That these diagrams and documents are open to numerous (mis)interpretations goes without saying, but because the logic design team probably uses a different set of design tools, the original models and test data are hard or impossible to recycle. Too bad the two groups couldn't find a way to work together more closely so that they could exchange computer models and data instead of paper.

As the detailed logic-design process unfolds and design responsibility moves to the logic-design group, focus shifts from system-level issues to the system sub-component implementation, with individual logic designers taking responsibility for the various pieces. A major, but inefficient, transfer of design information ensues. As part of this detailed design task, management assumes the logic designers will take steps to ensure that sub-component verification is carried out and is usually satisfied with observing that there is a block of time allotted in the schedule for this task.

That the verification task must provide proof of correct system-level functionality, beyond proving sub-component functionality, is not something management tends to consider. After all, the system architects proved the system works. The focus now shifts to ensuring that the sub-components work. Wouldn't it be nice if there were a golden set of models and test cases that could be used to check that the original specification and the new implementation were identical at some level of detail?

The logic designers, on the other hand, tend to see verification as something they do after they complete other design tasks such as design entry and synthesis. After all, in the short term, their jobs depend on producing lots of gates for their managers. That is an easily measurable productivity metric. That all these gates are functionally correct is something that generally won't be provable or measurable in a management sense for quite a while. That means it can be set aside until later. When logic designers start verifying their design functionality, they focus on their piece of the design pie using their personal understanding of the proper behavior based on the block diagrams and specification documents provided by the system architects.

Figure 1. A top-down HDL-based design should be composed of the elements above.

As the various sub-components start to come together, it usually occurs to someone that the sub-components should be connected to see if they work together. Of course, this is a tough time in the life of the project to be adding a major new task. Everybody has something else he or she should be doing. And this new task is not an insignificant piece of work. It will take months to create the system-level model, enumerate and write a reasonably complete set of test cases, run them, resolve the inevitable problems, decide who should fix them, and, finally, rebuild the simulation models. It probably need not be mentioned that at this level of modeling detail, computer run times will be horrendous. Too bad someone didn't think to do the preparatory work of creating the system model and preparing the test cases up front. Where are--what are--all those tests the system architects used?

The result of this lack of focus on system verification until near the end of the project is that there is little time and fewer resources available to support a coherent effort to thoroughly test the whole system. As a result, only a few, if any, system-level tests are applied to see if all the components work together as intended. Rather, management has to be satisfied with the haphazard collection of subsystem tests prepared, usually under time pressure, by engineers who may not fully understand the big picture.

Is it any wonder that while ASICs and FPGAs work properly during simulation, less than 50 percent actually work in the system they were intended to be part of? The bottom line is that if the design team doesn't adequately check out design functionality, chances are better than 50-50 that a second design pass, or maybe more, will be needed to get the hardware working as intended.

Verification Design verification today tends to take one of two forms: (1) extensive, time-consuming software simulation of computer models; or (2) large, elaborate hardware prototypes using hardware simulation based on programmable logic. Both claim important advantages, but both are solutions to a common problem--design complexity.

Computer systems do thousands of things with endless possible combinations, sequences, and relationships between various autonomous subsystems. It is impossible to enumerate all the possibilities, let alone manually create a complete test suite. And, adding insult to injury, this exponential increase in the number of tests needed has propelled design verification from a reasonably small, straightforward part of the project schedule to a devilishly complex monster that can consume two-thirds of the project schedule.

The strategy Then what constitutes an effective verification strategy--one that maximizes the chance of achieving a one-pass correct result? In short, the answer is top-down HDL-based design. The following elements, illustrated in Figure 1, must be accounted for in top-down HDL-based design:

Concept This is the original idea--the design requirements and feature set--that is generally a document.

Design data This is the design data in its various representations (views). Each description serves a specific purpose. There are five models of the computer system that are widely accepted as useful to the design process. Each model describes the design in progressively greater detail.

Performance models Performance and behavioral models are key to top-down design. They replace design-specification documentation and together represent an executable specification (simulatable) of design intent. The performance model allows system architects to consider "big-picture" performance issues without being burdened and distracted by the massive detail involved in detailed logic design. Performance models account for issues such as throughput and latency, and they define process or operation timing relationships. Actual data processing has no place here--these models do not calculate data values, although they can define data transfer paths.

Behavioral models This model defines computer system data-processing algorithms. System architects use this sort of model to consider big-picture functional issues, again avoiding the detailed logic design. True behavioral models contain no clocks, no buses, no data registers or pipelines, and no explicit state machines. Neither do they contain any information that suggests a preference for a particular implementation out of the many possible correct hardware implementations that may exist. Currently available synthesis tools don't work well with these models because there aren't enough clues to a desired implementation to allow inference of an acceptable implementation. Without explicit definition of the system clocks, buses, data registers or pipelines, or state machines, synthesis tools are hard-pressed to create anything useful to the design engineer. There are just too many possible implementation choices.

Behavioral models also provide a basis for developing the system-level verification test suite. Testbenches developed from the verification plan and verified at this level embody the intent of the system requirements. Later, they can be recycled and used to verify derived synthesizable and gate-level models. Because the testbenches are "known good," they can mean the difference between first-pass design success and never-ending design reiteration. What's new is that these models and tests are no longer the private property of the system architects, but an accessible, understandable, integral part of the design data available to the entire design team.

Synthesis models Synthesizable models are used to define a specific hardware implementation of the computer system described by the performance and behavioral models. They are not a re-coding of the behavioral model. Synthesizable models are everything a behavioral model is not. The model contains clocks, buses, data registers or pipelines, and explicit state machines. In other words, this model describes the same functionality as the behavioral model but in greater detail. Where a behavioral model may be a few thousand lines of code, synthesis models contain tens of thousands of lines of code.

Synthesis models are primarily used to drive synthesis tools. They aren't good for much else. Compared with behavioral or performance models, synthesis models are hard to read and understand, and they take forever to simulate. In an ideal world, no one would waste time coding these models. Unfortunately today's synthesis tools are not up to the task of synthesizing real behavior models into acceptable results, but they help engineers avoid complexity by shouldering the burden of detailed gate-level design work. Although synthesis models are complex, they are considerably simpler to construct than the gate-level models that are the next step in the design process.

Gate-level models and actual hardware Not much point in spending time explaining these. Gate-level models are either generated by a synthesis tool or written by hand and are used to drive the physical design process. And actual hardware (functionally correct) is the goal of this whole exercise.

Verification plan This is something new. This short and concise document states how the design will be tested to ensure that it meets all design requirements.

The verification plan should answer the following questions:

• What requirements or features should be tested, and why?

• What can't or won't be tested, and why?

• How should each requirement or feature be tested: via inspection, via simulation, via
  formal methods?

• When, during the design process, can each requirement or feature be tested: during performance or behavioral modeling, during synthesis code development, etc.?

• What version of the design representation will be used to verify each requirement or feature tested: at the system-behavioral level, at the synthesizable-component level, at the system gate level, at the gate-component level, etc.?

• What sort of test will be used: exhaustive testing, selective testing, random-pattern testing, etc.?

Keep this verification plan short, and keep it at a high level. This document will be (or should be) reviewed by many people with an eye toward getting everyone to agree that the proposed testing is complete (no holes). Only a readable document will ensure that such reviews occur. Historically, verification plans have been disorganized and undocumented. They tended to be haphazardly stored in the minds of project engineers. Because of increasing design complexity and the exponentially increased complexity of testing, the luxury of such sloppiness is no longer tolerable.

Testbench suite This is the implementation of the verification plan. It includes all the testbenches developed as part of the verification process. Using the simulation-based verification tools described below, these testbenches constitute a method of dynamically exercising the hardware design. Testbenches are targeted at various phases of the design process or various parts of the design hierarchy. Some testbenches are applied at the system level; some are created to test synthesizable component-level or gate-level models; and some are reusable at multiple levels of the design representation. Test suites are often generated with the help of automated test-suite development tools.

A specific testbench consists of a copy of the unit under test (UUT), a routine to generate UUT stimulus, and another routine to check that the test completed successfully (also known as a self-checking testbench). UUT stimulus is created either by some routine executing an algorithm or by reading in patterns from an external file. Algorithmically generated tests may be targeted test sequences or randomly generated test sequences. Checking routines may be as simple as a comparison against predefined "golden" expected results or a complex prediction scheme that observes input stimulus and independently predicts expected outputs for later comparison against UUT results. For more information about creating testbenches, see "Test Bench Tutorial Part I" in the April 1995 issue and "Test Bench Tutorial Part II" in the May 1995 issue of Integrated System Design .

Simulation-based verification tools Software simulators (event-driven or cycle-driven), and hardware simulators (accelerators, emulators, and real-chip modelers), dynamically exercise the UUT and predict how the actual device should respond based on patterns supplied by a testbench.

From a verification perspective, engineers are interested in simulation speed and predictive accuracy--but speed and accuracy are mutually exclusive. Speed is inversely proportional to design size and design representation detail. Behavioral modeling, hardware accelerators or emulators, and cycle-driven simulators were all developed, at least in part, to reduce simulation run times. Accuracy is a function of design detail and is proportional to the detail in the model--if a model contains no circuit-delay information, it makes no sense to look for detailed timing behavior in the results of the simulation.

Simulation-based verification tools are also dogged by the question of how much simulation is enough. Verification coverage of the design feature set is only as good as the testbench suite. To the extent that the testbench suite doesn't test all features, design correctness is at risk.

Formal methods-based verification tools Oversimplifying somewhat, these tools statically compare the logic functions of one model against the logic functions of another. If the individual functions are identical, the models must be identical. The tools come in two flavors: equivalence checkers and model checkers. Equivalence checkers are much more widely accepted and used in the industry today.

Equivalence checking attempts to reduce or eliminate the need for simulation of derived models such as synthesis or gate-level models--but not the original top-level model. If a synthesis model is statically equivalent to a gate-level model, there is no need to simulate the gate-level model to verify functional correctness. Eliminating the need for extensive gate-level simulations can reduce project schedules significantly.

Of course, this says nothing about whether the synthesis model is functionally correct. If the synthesis model has a bug, and if the synthesis tool accurately reflects the bug in the gate-level model, then the equivalence checker will dutifully declare that both models are identical. Model equivalence also says nothing about the timing characteristics of the derived model. Other methods beyond the scope of this article must be used to verify timing specifications.

Probably the greatest deficiency of currently available equivalence-checking tools are that they only work well in cases where the two design representations are already similar--for instance, designs where each has an identical set of memory elements (a synthesis model and the derived gate-level model). These tools fall down when the models differ greatly in structural content--a behavioral model and a synthesis model, for instance.

Model-checking tools attempt to reduce the need for simulation altogether. Such tools use some sort of high-level expression of design intent, such as a set of design assertions, and "prove" that the detailed implementation of the actual design is equivalent to the assertions. Unfortunately, creating the high-level assertions can be just as time-consuming as creating the more detailed implementation. And what about the question of how many high-level assertions are enough?

Conclusion New verification methods and tools, based on static-analysis techniques, are providing clear direction about the future of digital design. Manually created testbenches are being supplemented by tools that automatically generate tests; event-driven simulators are being replaced by cycle simulators; simulation is being supplanted by formal methods; and simulation-based timing analysis is being displaced by static timing analysis. In each case, the newer technology demands that the design description be less detailed, less customized, less complex in nature and easier for the tools to understand.

Traditional verification strategies are choking on design detail. New verification tools and methods are predicated on the assumption that detailed design descriptions are unnecessary because design information can be tuned to avoid tool limitations, and play to the strengths of new techniques. Engineers must move toward imposing stringent design rules, such as strict clocking schemes, on the logic-design process. Such design rules allow the tools to make simplifying assumptions while maintaining predictable accuracy. The rules also provide a basis for design verification that uses static analysis--a less time-consuming process then simulation-based analysis.

Larry Saunders is one of the principals of SEVA Technologies Inc. He can be contacted at (619) 538-6283 or lfs@sd.seva.com.

To voice an opinion on this or any Integrated System Design article, please e-mail your message to michael@asic.com.

integrated system design  April 1997