During the design of a fifth generation SAS device at LSI, it was clear that our testbench needed significant updates to verify the new features. The environment surrounding the SAS expander design had become cumbersome to manage. With each generation many new features had been added to the design. The new features complicated the existing environment, which, although flexible, had not been designed to verify these features. And, as each new feature meant adding directed tests to the library, after four generations the library contained thousands upon thousands of tests.

Of course, vital verification knowledge was embedded in the existing test patterns that we wanted to preserve. However, documentation associating the tests to the relevant sections within the design specification needed improvement and porting the tests directly would also involve many tedious hours (that our team did not have). As if all of these factors weren't problematic enough, all of this work would need to be repeated for future generations of the device.

Clearly, we needed a new approach; one that could encapsulate the verification knowledge in a portable, documented form that could move easily through future generations of the project without manual intervention. The new approach also needed to show which features of the device had been verified —not just a list of directed tests that had been written.

We used the Open Verification Methodology (OVM) with verification management and functional coverage to satisfy these requirements. Verification management allowed us to track and analyze coverage and test data, and functional coverage gave us unequivocal data on what had been exercised in the design. By focusing our efforts on functional coverage measurement development, we eliminated the effort of porting an extremely large number of tests.

Verification technology has taken a leap forward with the introduction of the OVM and more sophisticated, effective coverage devices —in both the form of SystemVerilog assertions (SVA) and verification IP (VIP) that has built-in coverage features. Using these new technologies allowed us to focus on the real task of verifying the design rather than the administrative overhead of managing and maintaining a library of directed tests. These new technologies are far less sensitive to specification changes than older directed test environments as they allow for predominantly modular verification environments. This is especially useful in storage designs, like ours, where market pressures often lead to specification changes late in the design cycle.

We split the team: half of the team focused on understanding the existing directed tests and converting these to coverage measurements, assertions, or monitors. The other half of the team developed the new verification environment.

The team capturing the information from the directed tests needed an auditable way to ensure that all of the information had been captured. The first step toward doing this was to take each directed test and translate it into either a cover group or cover directive. Some verification tasks could not be translated into SystemVerilog coverage constructs. These were added to the scoreboard requirements for the new OVM-based verification environment. For the very few cases where a test could not be translated into SVA or a scoreboard check, the directed test was ported to the new environment. Our verification management tool enabled us to use a spreadsheet format to directly link each legacy directed test to the SystemVerilog coverage construct created to replace it. The team could then run the old environment against this to make sure that when the legacy tests ran they triggered the coverage feature, this gave a link between the new coverage routine and the old tests. Once this work was complete, we were confident that the verification information had been extracted from the old tests and that a clear link existed for review between the old tests and the new coverage metrics. At this point the legacy environment could be abandoned, and we could move forward with the new coverage methodology. This also justified to management the effort our team spent on this activity, as it was no small task and took many months to complete.

The following is an example entry of the spreadsheet used to keep track of the testcases as they were converted to functional coverage measurements. Each engineer would keep a spreadsheet for the testcases that he or she was assigned. All the XML files were then merged together into one final spreadsheet that was fed back into the verification management tool. The second column of this spreadsheet lists the name of the pattern that was converted while the fourth column lists the functional coverage measurements replacing it. Once all the testcases were converted to functional coverage constructs, we were able to call this task complete and move all subsequent coverage development work over to the new OVM testbench environment.

Click on image to enlarge.

The next step in the process was to develop a verification plan for the new OVM environment. This was developed around the features of the design that needed to be verified, based on the architectural and engineering specifications. A list of SystemVerilog coverage points, scoreboard checks, and a few directed tests were generated from the plan, helping identify any OVM testbench components we had not thought of previously. This formed the basis for the architecture of the new environment.

Each section of the new test plan is linked to the cover groups, cover directives, code coverage, and directed tests that are required to verify that feature, giving a clear picture of verification progress against the design and architectural specification once a regression has been run.

Managers also gain far greater insight into the verification process. Instead of seeing only how many directed tests have been written out of an arbitrary total, it gives a real view of progress against the required design features. It also helps avoid the schedule surprises that happen with directed environments, which often occur when engineers think up new tests as they write tests in unrelated areas, moving the "verification done" goal further away each time. The following is a sample entry from our test plan.

Click on image to enlarge.

The output from the verification management tool, showing the verification progress of each feature, gives engineering managers a clear indication of the risks of an early tape out. It also allows an easier assessment of the impact of specification changes. The verification management tool also allows each engineer's progress to be tracked, allowing resources to be moved to critical areas that may stall the project. Supporting team work, not to mention reuse, our new approach allowed engineers outside our core verification team to review the test plan because it was written in terms of the design features they knew, rather than an esoteric, legacy verification environment. We ordered the features with section numbers identical to the EDS and architectural specifications. This provided proof that we had verified the design exactly according to its specification. It also will give us the ability to quickly answer common future questions of the ilk: "Is there a coverage point that represents the feature described in Section 3.3 of the EDS?" Expanding the verification reviews by making the documentation accessible to other engineering groups was an unexpected benefit of this process. For example, software engineers can use it for validation of their code and firmware engineers for validation of the firmware.

As a result of our new approach, test plan spreadsheets exist that describe the coverage points for lower level modules. From this point on, the coverage constructs and test plan associated with each reused section of our design can move with it.

Click on image to enlarge.

The Verification Management tool makes it possible to track the impossible amounts of data generated in the verification process, included a window into coverage progress.

Although it is too early to make a full comparison, we are already able to predict that an order of magnitude fewer tests will be required to verify the design. Leaving behind the unnecessary baggage from the legacy directed test environment will provide enormous savings in administration and maintenance of future versions of this design and verification environment. And it will be far simpler to verify new features. Much of the existing coverage and stimulus should be tolerant of changes in the environment because of its modularity. Clearly, reusing VIP improves quality and saves time, improving verification efficiency and reducing respins.

Jana Richard LSI, Staff engineer

Dan Cohen Mentor Graphics, Application engineer