Design Automation

A New Metric for Determining Design-Vector Completeness

Intel looks at the utility of grading vectors for verification completeness.

by Leonard Drucker & Brian Vaughn

When is design verification complete? Even with today's powerful tools, this question has not been answered adequately. A new metric is needed--one that a new type of verification, design-vector grading (DVG), can provide.

DVG uses proven grading techniques, akin to those used in manufacturing fault analysis, to determine the completeness of design-verification vectors. It generates a metric that indicates the design percentage that has been functionally verified, and it also identifies areas that have not been thoroughly tested. This process exposes design flaws not found through other methods. Note that DVG's value is predicated upon functional operation, not scan operation.

A project team in the Intel Embedded Microcontroller Division's Automotive Group (Chandler, AZ), which developed the DVG methods presented here, recognized the merits of using the grading process in design verification. The motivation to create the methodology originated with the team's observation that some design flaws and test holes had eluded the design verification group. However, the flaws were detected during the fault-grading process, and after the design verification was considered complete. The following are examples of the design flaws or test holes we encountered:

Shift-byte A test was developed to verify all combinations of shifting a byte; however, it failed to include a shift-by-zero operation, which causes a special microcode branch. Therefore, design verification was incomplete.

Interrupt flaw This flaw manifests when the CPU has the bus, an interrupt is received, and the CPU does not release the bus and service the interrupt. During fault simulation, an engineer observed node faults in the interrupt area that were not detected. Analysis indicated that the CPU serviced the interrupt if the instruction queue contained data, but if the queue was empty, the CPU ignored the interrupt.

Although the methodology uses familiar fault-simulation tools, its application of these tools in the strategic placement of design faults is unique. DVG marks an important advance in the quest to realize functionally correct design.

Verification tools for design feedback Typically, design verification encompasses developing functional vectors, generating random vector sets to catch corner cases and, sometimes, HDL code-coverage tools and gate-level toggle testing.

Random vector generation is used with microprocessor and microcontroller designs. Instruction sequences are generated in the hope of uncovering design flaws that may not be intuitive. Unfortunately, this technique only focuses on one design aspect: program execution. It does not consider other chip functions that may be triggered by external events, such as interrupts, as evidenced by the interrupt flaw described above. Although random testing was performed, the flaw was not detected.

Fault simulation takes the concept of random testing one step further. Instead of randomly mutating the test sequences, the logical design is mutated. In mutating the design rather than the vectors, each element can be tested. The mutated design brings to light the testing's effectiveness and provides a higher degree of confidence that the testing is comprehensive.

Some CAD vendors offer code-coverage tools to help with the verification process. These tools report which HDL design code lines were executed because lines not executed are potential test holes. Code coverage allows a detailed analysis of the internal HDL design structure. Unfortunately, it only looks for the execution of a line of code, not code functionality. For example, a test for code implementing a multiplier could be as simple as zero multiplied by zero. Although the code-coverage tool would show that the multiplier code had been exercised, the function has not been fully tested. However, this gives the verification engineer some insight into the thoroughness of the verification vectors.

Toggle-coverage tools are used on gate-level designs. These tools count how many times a node in the design toggles (changes logic state) and they address the controllability of internal points in the circuit. The toggle-coverage tool will give the verification engineer some understanding of how thoroughly the design has been tested. If a design section's nodes have not been toggled, a test hole has potentially been identified.

But the inverse is not true. Just because all nodes in a module have been toggled does not mean the module has been tested adequately. For example, assume you have a 4-bit counter with a parallel-load function. All zeros are loaded into the counter, followed by all ones. At the end of this sequence, each of the flip-flops' output nodes will be toggled to "0" and to "1"; however, no counting was performed. Toggle testing is an inadequate method for determining the completeness of a verification vector suite.

While toggle testing provides controllability of a circuit's internal points, it does not provide observability. By contrast, DVG requires controllability and observability to declare a node tested, as opposed to controllability alone. The added observability requirement usually forces the design to undergo rigorous execution. In the counter example above, the DVG process would first insert a faulty condition onto one of the counter's nodes. The test-vector sequence would then execute, with the goal of creating a condition in which the fault-free node version has a value opposite that of the faulty node. The fault must then propagate to the external I/O to be detected. Because this is done for every node in the design, a metric can be declared as to how many nodes were and were not tested.

DVG methodology reduces grading effort To make DVG feasible in the design environment, the Intel project addressed three time-consuming areas: execution of design grading, analysis of results, and development of additional verification tests. This evaluation required analyzing the goals of design grading versus those of fault grading, which is traditionally used to detect manufacturing flaws.

Manufacturing flaws can be opens, shorts, or structural defects in the design. Therefore, the grading process must be comprehensive. Note that design correctness is implied. Design flaws, by contrast, are functional, with manufacturing perfection implied. Because functionality can be proved without analyzing every structural component, the grading effort can be reduced drastically.

The DVG methodology First, only node faults are graded, rather than input and output faults. This can significantly reduce the number of faults to grade, which, in turn, reduces grading execution. A fault is defined as an error injected into the design. A node fault is a fault placed on a net.

Second, the design is partitioned into three regions: high-level-function critical nodes, low-level-function critical nodes, and non-critical nodes. A critical node is one that is crucial to design functionality. Each region has specific DVG goals.

High-level-function critical nodes must be 100-percent design-vector graded. These nodes typically interconnect high-level blocks in the first one or two levels of a design. A high-level block may be a UART or an arithmetic logic unit (ALU). If the verification suite cannot detect faults on these nodes, the high-level functions have not been thoroughly exercised.

Low-level-function critical nodes must obtain a design-vector grade of 80 percent. These nodes typically interconnect low-level functions, such as counters and registers, inside high-level blocks. Detecting these functions is more difficult because they are buried in the design. An 80-percent grade trades verification completeness for grading and test-development times.

Non-critical nodes are not simulated or analyzed, because these nodes normally interconnect simple gates, such as AND gates. Though these nodes are essential, detecting critical nodes implies detecting these non-critical nodes. Consider a ripple counter, for example. Fault detection of the critical nodes, which are the primary counter I/Os, cannot occur unless the non-critical nodes are exercised. Eliminating these nodes from consideration reduces the design-grading effort.

Finally, reducing analysis time entails setting criteria for the faults that should be analyzed. For example, the Intel design team eliminated possible faults, clock faults, and enable faults from analysis. Although these nodes are crucial to design functionality, they are manifested as undetects of other critical nodes.

Using DVG can reduce the verification cycle by quickly identifing functionality where design verification is incomplete, as shown in the following example. This knowledge focuses verification activity.

Figure 1. The importance of a given hierarchical level determines the amount of fault coverage required by that level.

DVG example DVG can be illustrated using a 4-bit ALU test design developed as a simple simulation test for Zycad simulation accelerators (see Figure 1). When applied to the design, the DVG method detected two test holes in the simulation vectors in the first half hour.

The 4-bit ALU has three hierarchical levels: interconnecting four 1-bit ALUs; interconnecting the adder, decoder, and logic_operations blocks that compose a 1-bit ALU; and internal logic for the adder, decoder, and logic_operations blocks. Its inputs and outputs are as follows: a 4-bit A and B input, a 2-bit function (F) input, a carry-in (CI) input, a sum output, and a carry-out (CO) output. The functions that the ALU performs are as follows: A and B when F=3D0; A or B when F=3D1; A plus B when F=3D3. This design provides correct output when stimulated by its test suite. There are 64 vectors in the test suite (see Table 1).

The 4-bit ALU simulation results are correct when exercised with these vectors. But how much of the ALU design does the test suite exercise? Does it cover all possible functions? DVG can provide the answers.

The DVG methodology partitions the design into three regions and grades the vectors in only two of those regions. Only node faults are simulated, and only undetected faults are analyzed. The simulation tools used in this example are from Zycad Corp. (Fremont, CA).

The first region, the high-level critical function, is normally the first one or two levels of the design. In this case, because the circuit is simple, this partition encompasses only the top level, which consists of four interconnected, 1-bit ALUs. The DVG goal is 100 percent.

The process is to first generate a list of faults to simulate. This is done with the Zycad fog2mm tool. Note that the 1-bit ALU is declared as a fault primitive (the design's lowest level for fault insertion). Therefore, faults will only be inserted at the boundary of each 1-bit ALU block. Thirty-eight faults are inserted. Next, the Zycad fault-simulator tool is executed to grade the vectors. This tool produces an output file known as a Fault OriGin (FOG) file. This file shows the detection status of every fault inserted in the design (detected, undetected, possibly detected, etc.). The undetected faults are the ones of interest.

The DVG for this level is 97.4 percent. Only one fault is undetected on the primary CI bit. A quick test-vector examination shows that the CI bit was never set to "1." Therefore, the carry-in function to the ALU was not tested. Although this oversight may be apparent in this example, it could be overlooked if the 4-bit ALU were buried inside a real design. Test vectors were added to detect this fault. The new set of vectors achieved a 100-percent DVG.

The second region, the low-level critical function, is the second level in this example, and it interconnects the three blocks that compose the 1-bit ALU. The three blocks are the adder unit, the decoder unit (which decodes the function), and the logic_operations ("and," "or," "not") unit. The DVG goal here is 80 percent.

The same grading process that took place in the first region also occurs here. However, this time the fault primitive is assigned to the individual blocks (adder, decoder, logic_operations), not to the 1-bit ALU block. The number of faults inserted is 74. The fault-simulator tool is then used to grade the vectors. The DVG is 97.3 percent, with only two undetected faults. Although this grade is higher than the goal, the undetected faults are still investigated. These faults are on the enable signals for the adder block of bits 1 and 2 of the 4-bit ALU. The test suite inhibits the propagation of these faults to the external pins. To realize why, you must understand the faults' effect on the design.

The faults cause the adder function to always be enabled along with the function decoded in normal operation. The propagation is inhibited because the A and B input vectors are always identical. This means the adder-unit sum output will always be "0" because the A and B input bits will always be either "0" or "1" (0+0=3D0, 1+1=3D0 with a carry out). Under normal operation, the design propagates the output of all four functions ("and," "or," "not," "plus") through an "or" gate, with only one function enabled at a time. Because the plus function always outputs a "0," which is not the controlling input of an "or" gate, the enabled adder has no effect on the overall ALU output. To alleviate this condition, the A and B vectors must be different. This will cause the plus function to output a "1," which is the controlling input to the "or" gate, at various instances. This will modify the decoded function behavior, allowing the error condition to propagate to the external pins.

Understanding this test hole required an investigation of the circuit. This type of error may not be detected in a normal verification process if the test vectors and design are not scrutinized.

The third region, the non-critical region, is not simulated because it consists of the internal logic for the adder, decoder, and logic blocks.

Design teams spend up to 50 percent of their cycle attempting to verify that their designs are error-free. Often, design flaws that go undetected by full design regression suites are uncovered during the fault-grading process. DVG, which uses the grading process, can play an important role for companies implementing a comprehensive verification methodology.

Leonard Drucker is principal applications engineer at Zycad Corp. (Fremont, CA).

Brian Vaughn is a product-development and quality engineer at Intel Corp. (Chandler, AZ).

To voice an opinion on this or any Integrated System Design article, please e-mail your message to michael@isdmag.com.

integrated system design  January 1997