Improve functional verification quality with mutation-based code coverage

Despite advances in stimulus generation and coverage measurement techniques, existing tools do not tell the engineer "how good" the testbench is at propagating the effects of bugs to observable points or detecting incorrect operation that indicates the presence of bugs.

As a result, decisions about where to focus verification effort, how to methodically improve the environment, whether it is robust enough to catch most potential bugs, and ultimately when verification is "done" are often based on partial data or "gut feel" assessments.

This article discusses the application of mutation-based testing techniques to measure and drive improvement in all aspects of functional verification quality for simulation-based environments as a solution to these problems.

Existing methods
Functional verification consumes a significant portion of the time and resources devoted to the typical design project. As chips continue to grow in size and complexity, designers must increasingly rely on a dedicated verification team to ensure that systems fully meet their specifications.

Verification engineers have at their disposal a set of dedicated tools and methodologies for verification automation and quality improvement. In spite of this, functional logic errors remain a significant cause of project delays and re-spins.

A key reason is that two important aspects of verification environment quality— the ability to propagate an effect of a bug to an observable point and the ability to observe the faulty effect and thus detect the bug—cannot be analyzed or measured. Existing methods such as code coverage and functional coverage largely ignore these two aspects, allowing functional errors to escape the verification process despite excellent coverage scores.

Code coverage at its core is a simple measure of the ability of the stimulus to activate the logic in the design, where "activate" means execute every line, toggle every signal, traverse every path or some similarly discrete activity.

While this is a necessary condition, you can't find a bug if you don't "touch" the code related to the bug—it is certainly not sufficient to expose the presence of all or even most problems in a design.

Code coverage says nothing about the ability of the verification environment to propagate an effect of a bug once activated or detect its presence assuming propagation is achieved. Verification engineers thus accept that while code coverage provides interesting data, it is a poor measure of overall verification environment quality.

Functional coverage is generally more interesting, necessary and useful in its own right. In simple terms, it provides a way to determine if you've exercised all important areas of functionality, where "important" is defined in various ways, such as "all operational states", "all functional sequences" or the like.

The rub is that, by definition, functional coverage is subjective and inherently incomplete. The functional areas (functional coverage "points") to be checked are defined by a set of engineers and typically based on the design specification—a document that thoroughly describes how a design should operate but does not provide a comprehensive view of how it should not operate.

If the specification considered all possible bugs that could exist in the design description and described how they might manifest themselves in terms of function, it would be a simple matter of translating this list into a set of functional coverage points to be checked comprehensively during verification.

Unfortunately, this is not the case, and while functional coverage provides useful and necessary data, you must have some means of determining if you are verifying the functionality laid out in the specification. Like code coverage, it is a poor measure of overall verification environment quality (Figure 1 below).

Figure 1: While functional coverage provides useful and necessary data, you must have some means of determining if you are verifying the functionality laid out in the specification.