The forgotten SoC verification team

As system-on-chip (SoC) designs have become ever more complex, verification engineers have risen in importance to become an integral part of a product’s success. With more SoCs being designed, there is a growing class of verification engineers who are woefully under-appreciated in terms of the complexity of the job they have to do and the lack of tools made available to them. These are the engineers given the responsibility for integration verification as well as system verification and validation.

This article defines the unique problems that SoC verification engineers face in their jobs and outlines an approach that provides a level of automation for them similar to that enjoyed by block-level verification teams. It also discusses longer-term implications of this approach within the overall SoC development flow and demonstrates that a higher level of abstraction is necessary for efficient and effective verification.

The unique SoC verification problem
High-end chips may contain 200 or more discrete IP blocks. Many of these blocks come about as part of the system design process, where functional needs are identified and those that do not offer product differentiation are fulfilled by an existing IP block wherever possible. This leaves the in-house design engineers to concentrate on the most differentiated functions: the high-value parts of the SoC.

What about the integration of these blocks? IP blocks can come from many sources, including internal design teams, development partners, EDA vendors and third-party IP providers. All these blocks need to be integrated with new blocks being designed for use within the particular SoC.

Some blocks may have been designed for reuse, which means that they likely will have several modes of operation or ways in which they can be used or configured, with only a few of these options actually used. Some are parameterizable and some may have been modified to fit the particular set of design requirements unique to the current design. Blocks may be at different levels of stability and quality. They may or may not come with testbenches designed to demonstrate that they function correctly in a standalone environment. Most testbenches are of little help when it comes to stitching blocks together and verifying that they can collectively perform a useful function.

This is the nightmare that the SoC integration verification team must face.

One of the reasons why SoC design is based on the divide-and-conquer approach is that it is too difficult for any one person to fully understand the total functionality. The same is true for verification because the task should be divided such that each aspect of verification can be performed by different people, possibly with different skill sets.

Redundancy between these verification tasks must be minimized. For example, there is no point in having integration engineers repeat verification that has already been performed at the block level. Each block has been verified in a standalone manner and thus, when integration verification is performed, the team should not attempt to do this task again.

Instead, the verification should focus on ascertaining that the blocks have been connected correctly and that collectively they can perform the necessary system-level functions. In addition, there are certain types of functionality that only become apparent and verifiable at the full-SoC level –– the ability to support concurrency, power and clock management, for example, and performance issues such as throughput and latency.

SoC integration engineers must perform three levels of verification, in addition to the block-level verification generally performed by different groups. Naturally this testing must be done at the “bare metal” hardware level, with no production operating software or drivers. However this article uses the metaphors of “drivers," “applications” and “performance” borrowed from operating systems.

The first level is the driver level where the integration verification team concentrates on the ability of blocks to effectively communicate with each other. This communication is most commonly between a processor and a peripheral, but could also involve DMA engines, memory subsystems or other infrastructure blocks. This level establishes that the necessary communication paths are functional.

The second level, which builds on the first, is the application level. Here the focus is on verifying complete paths that represent application use cases for the SoC. Verification engineers ascertain whether the SoC can perform the necessary functions, whether multiple paths can be exercised concurrently, and whether there performance bottlenecks.

The third level is the functionality that only exists at the system level, such as power and clock management. While pieces of the functionality are distributed among the blocks, the main controllers reside at the full-SoC level.

Collectively, these three verification levels make up the integration engineer’s responsibility, and information necessary to perform each task is somewhat different. The verification approaches are also different from those of block-level verification.

Defining SoC verification tasks
While there is variability in the way that companies approach SoC integration verification, they probably include many of the tasks in the following list, each designed to answer a particular set of questions. These tasks represent a refinement of the previous three levels of integration verification:

• Connectivity. Can each processor access all of the memories and other peripherals within the system? Does everything get reset to the right state? Are the fields within register words mapped correctly? Is the memory map correct? This is part of driver-level verification.
• IP integration. Can a processor read and write to all of the registers of an IP block? Can DMA move data in and out of each block and are the correct interrupts generated on completion? If there are any GPIO pins on the chip, are they multiplexed correctly? This is also part of driver-level verification.
• Use cases. Can data move through the system correctly to perform defined tasks? This may start with single tasks and extend to multiple concurrent tasks. This is part of application-level verification.
• Performance testing. Under worst-case conditions, does the system perform with no loss of fidelity, dropped data, or other conditions that may cause unexpected degradation of system functions? This is another part of application-level verification.
• Stress testing. When multiple random tasks are initiated, can the system handle concurrency, maintain data coherence, deal with multiple interrupts, and retain data integrity? This is the closest verification category to techniques used at the block level. This type of verification may also be used for power and thermal analysis of the chip. This is a further part of application-level verification.
• System management. Do system-level features such as clock and power management function as intended? Do changes of configuration at this level affect the results of any of the other verification tasks? Typical scenarios trigger transitions in the system power states and execute IP operations to verify that blocks are behaving correctly under these conditions. This is system-level verification.

While this list represents some of the tasks expected during integration verification, most verification teams do not move beyond connectivity and basic integration verification. As shown in Figure 1, such teams address only the tip of the iceberg, leaving the more advanced SoC verification tasks underwater and unexecuted.

Lack of specialized tools
SoC integration verification is usually performed with the same techniques used for block-level verification. Historically, verification was performed using a suite of directed tests, each specifically hand-written to verify a particular feature or aspect of the block.

Over the past decade, many companies have adopted constrained-random verification using languages such as SystemVerilog or e. This allows multiple tests to be created from a set of verification models and enables aspects of the block to be tested that may not have been explicitly considered. Only when constrained-random stimulus fails to verify specific or required cases are directed tests written.

However, as size and complexity of designs grow and as engineers attempt to use constrained-random techniques for SoC integration verification, they are finding it to be less successful. Simply put, constrained-random is not effective at exercising a specific function.

In addition, constrained-random stimulus is a “shotgun” approach where it is difficult to get focus on any particular aspect of a design. Verifying the complete functional space requires redundancy, which translates to an inefficient process.

The inefficiency and ineffectiveness of the constrained-random approach is contributing to the SoC verification process growing out of control.

One problem is that sequential constraints or guidelines are difficult to describe in the existing testbench languages. The Universal Verification Methodology (UVM) standard provides guidelines for describing sequences and layering these into higher-level sequences. However, it is difficult to control dependencies between sequences, limiting the effectiveness of SoC-level testbenches. In addition, the UVM does not address embedded processors within the SoC; testbench languages were designed for transaction-based verification of hardware, and not the control of multiple pieces of concurrent software.

Further, it is difficult to fully exercise a large design by manipulating only its inputs, whether via hand-written directed tests or constrained-random stimulus. Software running on the embedded processors plays a critical part in the functionality of the SoC. This argues for a new method that encompasses both the code running on the SoC’s embedded processors and the testbench connecting to its inputs and outputs. This method must be able to handle multi-threaded processes running on multiple processors within the SoC and coordinate multiple concurrent activities within the chip and with the testbench.

Ideally, the new approach should be able to span all the SoC integration verification tasks shown in Figure 1, from the tip of the iceberg to its deepest point.