Design Tools

Newbridge Networks Adopts a Top-Down Verification Strategy To counteract the verification slowdown caused by increasingly complex ASICs, Newbridge used a top-down "spec-level" verification strategy for the first time and was impressed with the results. by Dan Schumacher

Managing the complexities of designing and manufacturing computer networking systems is no easy task. At Newbridge Networks, one pivotal area--the design and verification of ASICs--has been receiving increased scrutiny.

Newbridge has been delivering fully managed networks for transmitting voice, data, image, and video traffic for over 11 years. As a supplier of end-to-end networking solutions, our constant challenge is to keep pace with the market's demands by providing major technical advances and superior functional integration within narrowing time-to-market windows. In the process, we've accumulated substantial experience in the design of hardware and software for standards-compliant asynchronous transfer mode (ATM), time division multiplexing (TDM), frame relay, and X.25 networking systems. Here, as with most electronic system manufacturers, the pressure of meeting critical delivery dates within shrinking market windows has intensified as the size and complexity of ASICs have grown.

The most significant changes to our ASIC development strategy have come about because of the verification problems posed by system-level designs. In terms of time and resources, design verification can easily consume 60 to 80 percent of our design budget, and such a percentage is by no means unique to Newbridge.

Difficulties with verification
The problem, though, is not only how much it costs to perform verification, but also how much it costs when the verification process proves inefficient, inaccurate, or incomplete. Missed market opportunities that result from excessive delays or the need to respin silicon two, three, and even four times can cost millions of dollars in lost revenues. Approaching verification from a higher level of abstraction offered us some compelling alternatives, including significant improvements in the efficiency, accuracy, and completeness of our effort.

The verification slowdown ultimately led us to adopt a top-down, "spec-level" verification strategy company-wide, with Specman from Verisity Design at its center. We hoped that Specman would overcome, or at least minimize, the verification bottleneck arising in nearly all of our ASIC development programs (see "Specman Allows Top-Down Verification"). Under severe market pressure to verify two ASICs, we used the top-down verification strategy for the first time. That strategy enabled us to get the two ASICs working in the system in significantly less time than would have been possible using our previous methodology.

Improved verification efficiency is important, not only from a resource management perspective, but from a personnel perspective as well. Skilled verification engineers are extremely hard to find and, to some degree, their scarcity can be linked to the inefficiency of the tools at their disposal. At an efficiency rate of 40,000 gates per design, today's verification engineer is placed in a high-pressure, no-win situation when confronted with ASICs approaching half a million gates or more. Specman dramatically increases the verification engineer's efficiency, turning that no-win situation into a value-added contribution to the company's competitive edge.

Another benefit to performing "spec-based verification" with Specman is that it formalizes the process of communicating spec-level design and verification issues as information is passed from one engineer to another, which virtually eliminates the potential for the mismatched assumptions between engineers that can create architectural bugs in the design. By maintaining information at a higher level of abstraction, it's easier to capture, and because the information is readable by engineers, it's much easier to relate to the original specification.

Specman complements our use of hardware emulation by helping to find the great majority of bugs--both functional and architectural--early in the design cycle, allowing us to use the hardware emulator in its more effective role as a hardware/software cosimulation environment. Moreover, Specman enables us to create a spec-level environment first, before the design is even ready, and to run traffic on it to make sure it's running correctly at the highest level. Then, when the HDL design is ready, we can find bugs immediately, instead of waiting until after we've completed the design to create our test benches. In that way, we discover a majority of the bugs prior to hardware emulation, reducing the chance of getting bogged down by having to trace a long series of individual bugs, one at a time in the emulator. It takes much more time to get a design running in an emulator than in a simulator and even more time to debug it.

Figure 1

Verification methodologies

Basing verification on HDL checkers and generators and on hand checking resulted in low coverage. Twenty percent of bugs had to be found in hardware emulation and lab bring-up, resulting in many design respins (a). With the new methodology, a higher percentage of total bugs was found in stand-alone ASIC testing, leaving only 1 percent of bugs to be uncovered through emulation and lab bring-up (b).

Previous methodology
Our previous verification methodology began with the design engineers breaking the functional specification down to a set of modules, designing a test plan, and writing deterministic checkers and generators in an HDL, such as Verilog or VHDL, to exercise each module's design code. Besides being slow to code, incomplete, and non-self-checking, the deterministic tests were, by definition, limited by the engineer's ability to predict where a bug might occur. Generators created in such a manner provided negligible coverage of corner cases, and the checkers were never complete, forcing the engineers to go back through test logs to hand-check the results after each run. A further limitation of the approach was that it provided no ability to mesh or overlay HDL-based tests in an integrated verification environment.

Because of the limitations of HDL-based tests previously mentioned, we were only able to find 80 percent of the total number of bugs with our ASIC test benches (see Figure 1a). Even as we continued to write tests, we didn't find any new bugs. We hit a plateau because we didn't understand fully what was tested and what wasn't. In addition, it was hard to correlate our bug rate with the new tests we wrote. Consequently, we were forced to find approximately 20 percent of our design bugs through hardware emulation or, worse, through lab bring-up. As a result, we had to go through several design respins. In addition, without good verification metrics, department schedules controlled release dates.

Having reached our code coverage goals, the engineers performed hardware emulation on a prototype system. Although hardware emulation is extremely useful to identify the last few percent of the bugs in a design, if used too early it simply bogs down the verification effort. It's simply too cumbersome to step in and out of the hardware emulator to fix a long succession of bugs one at a time. The verification "dead time" created by that approach was far too great.

Coverage
Besides its inherent inefficiency, our previous methodology lacked adequate functional coverage, making it a major cause of multiple ASIC respins. Achieving 100 percent code coverage was a good early metric to verify that all portions of the HDL code had been exercised. However, it provided no guarantee that all facets of the functional requirements had been met. There were multiple paths and conditions affecting the function of a given area of the design that remained untested even after 100 percent code coverage was achieved.

To ensure satisfactory functional coverage, thereby reducing the potential for multiple respins, we needed the ability to identify holes in the functional coverage early in the verification process and to eradicate as many functional and system-level bugs as possible before going to silicon. Functional bugs, the more straightforward of the two, occur when something in the design has been implemented incorrectly. System-level, or architectural, bugs are more complex and more ambiguous in nature. They can occur as a result of a misunderstanding of the specification, an inaccurate functional definition, or mismatched assumptions between engineers working on separate but interconnected modules of a design. Deterministic tests--even at the system level--are poor at finding such system-level bugs because they require you to "predict" the communications breakdowns.

Specman delivers
Our first opportunity to use Specman and evaluate its effectiveness arose when the first silicon for two ASICs verified with our old methodology arrived back from the fab with numerous bugs. The ASICs, comprising approximately 150,000 gates each, were headed for a 10/100-Mbit/s switched Ethernet system that was already late to market, so we had to be aggressive. The ASICs ran at a clock speed of 40 MHz, incorporated a number of internal queues and shared buses with proprietary protocols, and both interfaced to a standard medium-independent interface (MII) bus. They would provide a comprehensive verification challenge that would test virtually every feature of Specman.

With our new verification methods, the efficiency of Specman made the rate of test generation much higher (see Figure 1b). Also, the superdirected and on-the-fly test generation made it easier to flush out our design bugs. Furthermore, using a combination of line and expression coverage, as well as functional coverage, we were more capable of writing new, meaningful tests. We could now uncover more than 90 percent of our bugs in ASIC stand-alone testing (through either module or full-chip testing).

We now had the bandwidth with our existing staff to write system-level tests in which we could further verify the ASIC in a system-level environment. That practice flushed out new types of design bugs, such as handshaking, interface, and initial specification errors. In the testing, we uncovered a significant number of the remaining bugs--about 9 percent. When we started hardware emulation after using Specman, we thus had a much more mature design and found very few bugs. Those that we did find were generally in the corner cases, requiring millions of simulation cycles to uncover.

As is most often the case, the major problems encountered in the ASICs stemmed from the inadequate functional coverage of our old methodology. The design engineers had successfully met their code coverage goals, highlighting major holes and performing most of their checks manually. The checks were not self-testing, because that would have required more time and resources than we could apply. Ultimately, the ASICs didn't work in the areas we didn't test.

Testing again
Starting from scratch and using two verification engineers who had no prior exposure to the original specifications, we created an architectural model of the designs (including a proprietary bus architecture) within Specman and automatically generated bus protocol checkers, bus traffic generators, and self-checking "superdirected" random tests in both an ASIC stand-alone and a full-system environment. Superdirected tests are pseudorandom tests, but they offer the user much more control. Whereas significant effort is required to drive the pseudorandom test generation to valid cases, Specman enables the designer to steer Specman's superdirected tests to corner cases. The superdirected random tests permitted our verification engineers to thoroughly test corner cases in a fraction of the time that would have been required using hand-coded directed tests (see "Verification Test Types: Working in Parallel").

Using our design specification and functional test plan, we were able to define numerous functional test points with the goal set at 100 percent coverage. Using Specman to define functional coverage enabled us to do a feature-by-feature validation of the specification and to test the interaction between modules. As we ran the tests and found bugs or functional coverage holes, Specman allowed us to quickly create new tests to cover the holes, and we quickly closed in on working designs. In all, the entire process took approximately 18 weeks, including three days of training at Verisity, three weeks of working with our applications engineers to accurately define the bus protocol, and three weeks of getting the verification engineers familiar with the architecture of the ASICs.

Verification Test Types: Working in Parallel

There are three categories of verification tests (see the figure). Directed tests (a) are generally created from the verification document, which is driven by the functional designing specification. An enumerated list of design features drive the directed-test generation. The directed tests are written to test one distinct feature of the design and are then run sequentially against the design to verify the functionality as defined by the functional specification. The problem with directed tests, however, is that the interactions between blocks are poorly tested. To better uncover corner cases, where errors creep in, many designers focus on pseudorandom testing, in which the designer randomizes the running of directed tests. The tests are generated by taking directed tests and mixing them across each other. Because a true random approach would run into many illegal sequences, a pseudorandom approach is used. The designer refines the tests further by running them in parallel rather than just sequentially. Running the tests sequentially (b) really changes only the directed test's initial conditions (running with a FIFO full or empty, for example). Running the tests in parallel, on the other hand, in which the tester mixes the tests across each other (c), better uncovers module interaction problems, forces interesting corner cases, and ultimately finds more bugs. Superdirected testing enhances pseudorandom testing by allowing the user to identify constraints and other conditions in addition to the concatenation of the tests. By mixing only the test cases that are relevant and including user-specified conditions, superdirected tests offer the best features of pseudorandom testing and high user control. Again, performing many combinations of test sequences in parallel (d) better uncovers module interaction problems. Whereas pseudorandom test generation requires significant effort to cover valid corner cases, the user can easily direct superdirected test to the corner cases.

Performance analysis
One of the essential questions facing every system designer is, Why is the complexity curve for our ASICs growing so rapidly? An obvious answer might be that as microprocessor speeds increase, the associated glue logic must also increase in speed and complexity to avoid creating performance bottlenecks. However, if we incorporate effective performance analysis at the architectural specification level, and if our specification more accurately reflects the actual implementation, we can eliminate a substantial amount of overdesign.

A side benefit to adopting a top-down verification strategy is that it brings the architectural assumptions and the actual chip design closer together, making high-level performance analysis easier to accomplish. Typically, our performance analysis team uses an ASIC's architectural specification to establish network performance goals utilizing a performance analysis tool, like Opnet from Mil 3, to model the overall system environment. Unfortunately, though the design intent is maintained, the actual chip design often strays far from the original architectural specification, leaving the performance analysis group effectively disconnected from reality. That's a fairly consistent problem throughout the industry.

Using Specman as the heart of a top-down design and verification strategy, we can model the chip architecture at the spec-level and create an effective bridge that reaches all the way to the gate-level design, enabling our performance analysis team to set more realistic performance goals for system-level throughput and latency. In that way, Specman helps establish greater continuity across the entire design and verification process and solves the disconnect problem for the performance analysis group.

Results
Our initial Specman project proved successful on all counts. We demonstrated the effectiveness of a top-down verification strategy within Newbridge, taking full advantage of the benefits afforded by Specman, and got the ASICs working in the system in significantly less time than would have been possible using our prior methodology. Although we can't know how many respins would have been necessary using the old approach, the efficiency with which our verification engineers managed the effort left no question that our return on investment using Specman was substantial.

We've already begun establishing dedicated verification teams throughout the company with the mandate to create a reusable verification infrastructure that uses Specman to implement a top-down verification strategy. Currently, eight different ASICs are being verified using Specman. Synergy between the design and verification groups is growing steadily, and the potential for verification reuse gains ground with each new ASIC.

We are also evaluating other verification technologies that may hold promise for future designs. They include cycle-based simulation and formal verification. In all, the momentum created by our new top-down strategy has taken us a long way toward overcoming the verification bottleneck.

Dan Schumacher was manager of design automation and services at Newbridge Networks, Inc. in Andover, Mass. (formerly UB Networks), when he wrote this article. He's currently the manager of switch and Gigabit ASICs at Sun Microsystems, Inc. in Chelmsford, Mass. He has over 16 years' experience in the ASIC industry and has worked in process development; test; process characterization; analog, digital, and mixed-signal design; full-custom, standard-cell, and gate array design; microprocessor and network architecture; and CAD tool development, support, and design methodology.

To voice an opinion on this or any Integrated System Design article, please email your message to miker@isdmag.com.

integrated system design  June 1998