design flow

Automating Functional Design Verification Sun creates a flexible, reusable multi-ASIC and system verification environment with a single engineer and proves its worth on a workstation graphics subsystem. by Allen Harris and Derek Pappas

Most ASIC design companies develop a new verification and test environment for each new project, designating at least one design automation engineer to develop the verification environment and write the programs required to automate it. When a project consists of a set of ASICs, organizations assign one design automation engineer to develop a separate ASIC verification environment for each chip and assign another to develop the system verification environment.

Limited resources for functional design verification led us to examine the verification process and analyze how to automate it for our requirements at Sun Microsystems' graphics and imaging department. We determined that the architecture, design, and development of a flexible ASIC verification environment is a complex problem. Our solution, named the Configurable Automated Verification Environment (CAVE), is designed to control single- or multiple-ASIC verification environments and also to port easily from one ASIC project to another.

A CAVE user commands complete control over the simulation parameters when testing the ASIC module, ASIC chip, or multi-ASIC system during a CAVE simulation session at any level of the design hierarchy. We fully automated the tedious tasks of locating tests, implementing the methodology, executing the proper design automation tools, and tracking the test results.

The Sun Elite3D project used only one engineer to develop the automated verification environment, rather than the four different design automation engineers usually needed to develop three ASIC and one system-level verification environments. Moreover, the single developer of the CAVE system spent considerably less time than did the multiple verification environment developers of previous single-ASIC projects.

Exploring the CAVE

The CAVE system was specifically designed to functionally verify the Elite3D graphics ASIC chip set and the future generations of graphics ASICs for the Sun SMCC High-Performance Graphics Group. The largest ASIC of the Elite3D chip set consists of approximately 500,000 gates; the smallest, about 200,000 gates. The CAVE system consists of a setup program (see "Arranging the CAVE"), a simulation/verification session program, a C/C++ behavioral simulator, a regression sequencer program, a system build program, and several configuration tables or files.

Verifying the chip set required us to verify each ASIC module and individual ASIC design. We tested the ASICs by verifying their interaction at the full system level in the different system configurations. The CAVE system was configured to automatically control and monitor approximately 19 different module- and chip-level test and verification configurations and 28 different system-level configurations, all of which execute independently of each other. Altogether, we ran over 1,600 tests to functionally verify the design.

We used a bottom-up verification methodology to functionally verify the chip set. We applied test stimuli to the inputs of the RTL and gate-level models of the design, then compared the RTL and gate-level results with the expected test vectors.

Our bottom-up verification approach consists of the following steps:

1. Test an ASIC module in its own environment to prove that the logic, control, and data paths are functional.

2. Test it in combination with its neighboring modules to demonstrate that the interface controls (the stalls and handshaking) are functional.

3. Test all modules at the chip level to prove that the ASIC works according to specifications.

4. Test the chip set in the different system configurations to prove that all the ASICs can function together as a system and to reveal ordering problems and protocol violations between them.

Simulating behavior with C++

The behavioral simulator is an integral part of our verification methodology. We developed the simulator using C++ because the object-oriented approach facilitates the design of reusable classes to be used on future simulators. Since compiled C++ programs execute significantly faster than RTL behavioral simulators, we can also use the same simulator as a platform for software development of graphics libraries and device drivers for the real hardware. We eliminated several gigabytes' worth of special test vectors for the ICs by providing test vectors on demand through the C++ simulator. When designers make minor modifications to the ASIC design, the system modifies the behavioral C++ simulator to reflect the design changes. As a result, any test vectors generated during subsequent simulation sessions will be modified accordingly.

Figure 1	CAVE simulation environment
CAVE automates a complex network of test benches, input and output files, and verification and simulation procedures. The system ultimately generates test summaries for use in further debugging the design.

Programs ranging from those for simple tests to highly sophisticated 3D models access the behavioral simulator as if it were the real Elite3D hardware. Test programmers access the behavioral simulator by reading and writing the actual Elite3D registers. Programming the behavioral simulator in such a manner makes it a source of simple or complex stimulus and response test vectors for testing the RTL ASIC design. The same tests can be used to test the ASICs during first-silicon testing.

We designed most ASIC tests to use the behavioral simulator to generate test vectors. The behavioral simulator can create stimulus-and-response test vectors at any defined test point in the hierarchy (a test point is a module or ASIC interconnect boundary where test vectors can be extracted). Each ASIC test point within the simulator has a switch setting that allows the output of test vectors at the corresponding point in the ASIC design, and the command line switches specify which test point vectors should be generated.

Since the Elite3D full system configuration contains over 200 test points, the CAVE system automatically sets each test point based on the level of the design hierarchy being tested. The CAVE system generates the command-line switches using multiple lookup tables during its simulation session.

Each ASIC designer developed a test bench for each hierarchy level he needed to verify. CAVE also can use test vectors generated by other programs to stimulate the behavioral C/C++ simulator. The latter are usually special vectors for testing
corner cases that the behavioral simulator normally can't generate when using the standard system level test paradigm. During a simulation test session, CAVE automatically selects the proper test bench for testing the RTL design; then it directs the behavioral simulator to generate the proper stimulus and response test vectors to test it.

Automating the environment

We analyzed the ASIC verification task we were assigned. Based on our analysis, we developed a verification methodology and automated the ASIC verification process using CAVE. In general, we treated each ASIC as a collection of modules and the entire system as a collection of ASICs. Though testing different parts of a single- or multi-ASIC system may require different test strategies, we did develop a generalized methodology:

1. Select a target unit to be tested.

2. Select a simulation mode of testing, such as gate-level or RTL.

3. Select a simulator, like Verilog-XL or VCS, or a hardware emulator from providers such as Ikos.

4. Select a test or a list of tests needed to test the unit.

5. Execute the tests or list of tests and output the test results.

6. Format and summarize the test results clearly and comprehensibly.

CAVE is a parameterized system based on a table-driven design and user configuration and global project configuration files. During the life of the project, the CAVE system source files are never modified. The designers and CAD and verification teams modify the parameters defined in the configuration files.

To facilitate the table-driven design, we gave each ASIC module, intermediate hierarchy, ASIC chip, and system an identical directory tree Several configuration files and tables specify to the CAVE system the test points at each level of the design. CAVE uses the information in those files and tables to specify the input and output test vectors at that point in the design hierarchy, the list of tests to execute for verifying the RTL design, the command-line directives to execute the behavioral simulator and RTL test bench, and other input parameters that control the testing at that level of the design hierarchy. The global project configuration files provide the location of the tests, behavioral C/C++ simulators, CAD tools, and other common files and programs. For each test executed, CAVE creates a test results directory that in turn resides in the working subdirectory and contains the generated .rc files, log files, and test vector files after a completed simulation session (see Figure 1).

One of the biggest problems in design verification is monitoring the progress of the required functional tests. Once a test is available, it needs to become part of the verification environment. The CAVE developer is constantly adding them to increase coverage of different parts of a design, making up-to-the-minute monitoring a critical task.

Master test list

The master test list file can track test development and ensure that the added tests become part of the verification environment. The master test list contains the names of tests that are currently being written, scheduled to be written, or currently available to the CAVE system. When tests are ready for use in the simulation environment, the designer marks the entry for that test in the master list as "available" for the CAVE environment to use. A single test can belong to more than one test list, and the master test list specifies each test list that a test belongs in. A script parses the master test list file and generates a test list file for each level of the design hierarchy automatically.

Scripting the build

We verified the Elite3D chip set with the CAVE system. The verification required the C/C++ behavioral simulator, 1,607 tests and diagnostics, several test lists, configuration files and tables, RTL and gate-level models, and test benches. In the process of verifying the chip set, we found and fixed problems in some of the many files and programs. An automated build of all the necessary files and executables occurred every night to ensure that the latest versions were available each day for the CAVE system to access. The build script--designed to build all executables and to check them for build inconsistencies or errors--provided a fail-safe mechanism to ensure that a validated build was available to the CAVE system.

After a successful build, the build directory becomes the release directory. If an error occurs during the build process, an e-mail message informs the CAVE system administrator of the problem or problems found. The CAVE administrator can then update and correct errors identified during an overnight build without interrupting the current on-line operation of the CAVE system. By giving CAVE access to only the release directory, we ensured that the system would see only verified executables. When tape-out milestones approach, the process-and its restricted access-plays a critical role in keeping errors from propagating through the rest of the verification environment.

Figure 2	Working directory
The user selects a working directory during a CAVE environment setup. During a session, the environment executes a test that drives the C/C++ simulator in the results directory, then sets Unix soft links in the RTL directory to reference the vectors generated in the Results directory.

We chose to speed execution of multiple simulation sessions with a pool of Sun Ultra workstations and Platform Computing's Load Sharing Facility (LSF) program. We built an LSF workstation pool of 100 Sun Ultra 1 and Ultra 2 systems executing simulation jobs in parallel.

To take advantage of these resources, we developed the CAVE regression program, which submits multiple CAVE simulation sessions to the LSF-controlled pool of workstations. The regression program submits parallel simulation sessions to the LSF batch monitor based upon the list of tests specified in the user test list file at that level in the design hierarchy. The regression program tracks the results of each CAVE simulation session, then generates a regression report designed to list failing tests at the top of the list and enumerate the reason for failures. The system formats and outputs the information into a regression status file that the user can then examine.

At times, the LSF workstation pool handled thousands of tests at once. Users were executing CAVE regressions for ASIC modules, multiple hierarchies, ASIC chip testing, and multi-ASIC system--testing all at the same time.

No CAVE-in

After our ASIC vendor delivered the new parts, we tested the parts in the system. We started by bringing up the ASICs with diagnostics and tests developed using the C++ behavioral simulator. The software window system was up and running within two days. We found only a few small design problems in the ASICs.

The three ASICs for the Elite3D chip set required 16 RTL design engineers to develop the RTL and gate-level models, and a team of 8 test verification engineers developed 1,607 tests to verify the designs and models. The behavioral simulator development team required 5 engineers, and CAVE required 1 full-time engineer for 6 man-months.

CAVE's table-driven design makes the system easy to port from one ASIC project to another. The main programs that drive the simulation sessions don't change, and the RTL source file project hierarchy must maintain the working subdirectory structure. Only the configuration files that drive the system require changes. Of course, for each new ASIC project, the team must develop a new C/C++ behavioral simulator that simulates the actual system.

Porting CAVE to the next-generation 3D graphics project, consisting of two ASICs, required just three weeks. On a previous ASIC design program, it took a design automation engineer the entire length of the ASIC design project to finish making changes to the verification environment.

The CAVE simulation environment

A typical CAVE Verilog environment performs a complete simulation and verification session. The user either generates or uses a default test list and sim file in the working directory (see Figure 2). The test list enumerates the sequence of tests to be executed; the sim file contains command-line directives to control the executable tests, behavioral simulator, Verilog simulator, and RTL test bench.

The graph file, written at the beginning of a project, defines the design hierarchy and the test points. CAVE uses it to locate the working directory that provides the imbedded default tables and files. The CAVE environment inputs the information to execute the selected test and generate the command-line directives, using .rc files for the behavioral simulator and Verilog interpreter.

CAVE then executes a selected test, which in turn executes the behavioral simulator and generates the vector files. Next, it executes the same test again, but this time it selects the Verilog interpreter, which executes an RTL test bench. The RTL test bench compares the expected test vectors with the output of the design under test, then stores the results in a results log file. A postprocessing script later filters the log files to produce a summary report.

A major goal of a verification environment is to verify the hardware design as soon as possible. Finding functional problems early in the design process makes them easier to debug and less costly to fix. Therefore we want designers to be able to verify their functional design using the test list for that level of the design hierarchy before checking the RTL code into the source database. The model builders then integrate the new RTL code and run regressions at a higher level in the design hierarchy.

We're currently enhancing CAVE's automated build script to allow automatic verification of an RTL model when any of the source files in the design hierarchy for that model have been edited. When CAVE receives a change to an RTL file, the build script will automatically start multiple CAVE regression sessions for the following levels of the design hierarchy affected by the change: the ASIC module, the ASIC intermediate hierarchy, the ASIC chip, and the system. This feature will allow us to track RTL design changes that work at lower levels in the design but may fail at higher levels. The system automatically submits regressions overnight and yields test results by the next day.

Disappearing test vector files

When running CAVE regression sessions, the LSF pool of Ultra systems generates files of test vectors to use to functionally verify the Elite3D design. When the pool is running several hundred tests simultaneously, though, the number of files generated for test vectors can become astronomical-100 Gbytes of disk space can easily become full under such conditions. Such situations become common during ASIC chip-level and system-level regression sessions for ASIC tape-out and AC sign-off milestones.

To solve that problem we're developing an RTL PLI to transfer test vectors from the C/C++ behavioral simulator directly to the RTL test bench, using a shared memory interface instead of files. We're also adding several multiple-CPU Sun Enterprise servers to our LSF pool to increase computing power.

Using shared memory and the RTL PLI, the system can transfer test vector data directly from the C/C++ behavioral simulator to the RTL model without generating any test vector files. This shortcut eliminates the need for large amounts of disk space required during heavy use of the LSF workstation pool.

Allen Harris is a staff software engineer in Sun Microsystems, Inc.'s graphics and imaging department in Menlo Park, Calif. In his 11 years with the company, he has designed much of the verification methodology for the organization, leading several teams of software engineers in developing C/C++ simulators for 3D graphics ASIC projects.

Derek Pappas is a CAD software and ASIC design engineer at Sun Laboratories in Mountain View, Calif. In his three years at Sun, he designed and developed the CAVE program, worked on the C/C++ behavioral simulator software team, and later worked as an ASIC designer. He previously spent three years with Intel as a design automation software and VLSI design engineer for the Pentium Pro and Pentium Pro Compaction ASIC programs.

To voice an opinion on this or any Integrated System Design article, please email your message to miker@isdmag.com.

integrated system design  April 1999