|
Product Evaluation: Verilog HDL SimulatorsThis is the third in a series of EDA product reviews that provides fair, comprehensive, and practical evaluations for our readers.By Yatin Trivedi and Larry Saunders
Since
last year's product evaluations of Verilog HDL simulators appeared (PC-based Verilog in April '94, page 12, and VHDL in July '94, page 12), we received a number of constructive comments from users and vendors. We have attempted to incorporate as many of them as possible in the new evaluation without jeopardizing the integrity and independence of the evaluation process. The "Inside HDL" column entitled "The Artistry (and Guile) of Product Evaluation" (March '94, pg. 56) still continues to be the guiding path
for these evaluations.
What's new? What have we changed for this evaluation? For starters, most users were interested in comparing Verilog simulators on workstations, and since most today are using it in workstation environment, we thought it was a natural adjustment to make this a Unix-based evaluation. This, in turn, created several significant changes in our evaluation process. First was the choice of the hardware platform and the operating environment. All seven of the participating vendors preferred to submit the version of their products that ran on SPARCstations. (This was also the platform of choice for users interested in comparison, we found, even though many of them use their Verilog simulators on HP, DEC, or IBM hardware.) SPARC-based workstations provided the common denominator for all vendors' latest products. Although most vendors also support other platforms and operating environments, they seemed not quite "up to date" on their latest release on non-SPARC platforms. The Sun Microsystems Computer Corp. (Mountain View, CA) SPARCstation 5 and Aries Research (Fremont, CA) Matrix 10 that we used as evaluation platforms were configured thus: the SPARCstation 5 was an 85MHz workstation with 128 Mbytes of RAM, 300 Mbytes of swap; the Matrix 10 was 55MHz, with 64 Mbytes of RAM, 170 Mbytes of swap. Both were operating under SunOS 4.1.3 (Solaris 1.0). The second change was in the choice of designs. According to most serious users, the designs chosen for PC simulator performance evaluation were too small. In fact, some commented that the size of the largest designs used in earlier evaluation were comparatively small and only "moderately interesting" in their respective design environments. So up we went, increasing the size and complexity of the designs. See Table 1 for a list of designs used in performance measurement.
Figure 1. The SEI weighs various aspects of the simulators' functionality differently, based on what we understand to be of importance to users.Thanks to some of our readers (and the legal departments at their companies) we were able, under non-disclosure agreements, to secure more actual designs than we needed. Among those who contributed the designs were Bay Networks (Santa Clara, CA), Sun, and Texas Instruments (Dallas, TX). We also changed the weights for the Seva Evaluation Index (SEI). Most readers commented that the key issue for them is the simulator's performance. That message came in loud and clear: they would be willing to cope with certain inadequacies and shortcomings as long as they could run large models at acceptable performance levels. The new weight for performance is 50 percent, and compliance is 20 percent. With this revision, the relative importance of other criteria obviously changed also. The new SEI appears in Figure 1. For this evaluation, we used the latest version of the simulators as of November 10, 1994. All of the vendors have had at least one release, possibly more, since then. Weighing in on the Seva Evaluation Index Before moving on to make comments on the individual simulators, we should make some comments relative to all the simulators, and on the SEI as it was used to evaluate them. Performance: (SEI weight: 50) A major factor in the selection of a simulator is its performance. Most serious users find simulator performance the major bottleneck in their design schedules. We measured performance in four parts: compile and link time, execution time, compile and link memory utilization, and execution memory utilization. We used nine different designs (see Table 1) for performance measurements. These consisted of RTL, gate, and mixed levels of abstraction, and each varied in size from small to large. We then compiled and simulated each design with a small, medium, and large number of vectors, and took performance measurements. The individual contribution of each design in measuring overall performance is shown in Figure 2. The designs used for performance measurements are actual designs obtained from readers. Since some of the designs given to us could not be simulated on all the simulators, we settled for the designs that gave the the fewest simulators problems, electing to use those that ran on most. Even then, some simulators had to be given zero points when they could not run the simulation correctly. Compliance: (SEI weight: 20) We ran more than 150 test cases to check the compliance of the simulator with the Verilog language. We considered Open Verilog International's Verilog Language Reference Manual (Verilog LRM), version 1.0, and Programming Language Interface (PLI), version 1.0, as standard. When in doubt, we also verified the results by running examples on OVIsim (Verilog-XL 1.6). Verilog-XL, VCS, and VeriBest have the best compliance with the OVI definition of the language. The other four simulators need considerable work to accept the full language and simulate correctly. Our evaluation focused on five parts: basic language, name space, gate and switch level, RTL, and hierarchy, but we gave the most importance to test cases in RTL and gate/switch categories. Some of these test cases are being submitted to OVI's compliance committee for public use. Debugging Features: (SEI weight: 10) We found that all simulators support third-party postprocessing tools such as Magellan (Systems Science, Palo Alto, CA), Signalscan (Design Acceleration, Campbell, CA), Undertow (Veritools, Los Altos, CA), and VirSim (Simulation Technologies, Minneapolis, MN) through VCD files ($dumpvars command). There is therefore no special discussion on the postprocessing capabilities for each individual simulator unless it is bundled with the product. We credited the simulators for any integrated debugging features (e.g.; source-level debug). Design Environment: (SEI weight: 6) The usefulness of a simulator extends beyond software simulation and into overall design methodology. We looked for interfaces to hardware accelerators, emulators, modelers, simulation backplanes, a SWIFT library, model encryption, ASIC library support, and interactive waveform displays. Verilog-XL provides the most hooks into different tools and libraries. VCS and VeriBest are close behind; the others need to put considerable effort in this area to make their respective products integrate well in a design flow. Unfortunately, we expect support for this category will be inadequate unless PLI is well-implemented. Since PLI is a separate category by itself, we decided that a weight of 6 here would be fair to avoid "double jeopardy." Programming Language Interface (PLI): (SEI weight: 6) Although many users frequently use the PLI, and most of the large
design projects depend on PLI-based verification, we found that overall support by Verilog simulation vendors was incomplete. Since the PLI 2.0 definition is not yet a standard and has not been implemented by any of the simulators, we considered PLI 1.0 for this evaluation. Verilog-XL, VCS, and VeriBest have the most complete implementation of PLI (tf_ and acc_ routines). Although the "binding mechanisms" are different, they offer identical functionality, so the ease of portability of PLI applications varied from one simulator to another. Essentially, though they take different roads (some more "scenic" than others), they end up in the same place. We ran 10 PLI tests in all, and removed the test case "hello.c," since all the simulators supporting PLI passed the test. Timing and SDF: (SEI weight: 6) Although most users favor static timing analyzers for timing analysis, a number of users depend on the specify block and SDF back annotation mechanism for timing analysis. Unfortunately, this is one area in which some of the simulators need the most work. We ran 22 tests for specify block which were not included in the language compliance category. We also ran five examples to check support for SDF back annotation. Technical Support: (SEI weight: 2) We lumped a number of issues in this category. None appears to be very important by itself, except when a user runs into trouble. We included documentation (Verilog language, PLI, SDF, users' guides), on-line help, the installation method, licensing scheme, training, and examples. All simulators come with good documentation of the language as well as a users' guide. All the vendors (either directly or through consultants) provided support via on-line bulletin boards, e-mail reflectors, ftp sites for downloading upgrades and patches, and basic training. The envelope, please... Well, enough about the evaluation process. Below is our take on each of the simulators. VCS 2.2 (Chronologic Simulation) Overall, VCS came out ahead of all the other simulators. It was the fastest simulator with the best overall performance score. Its language implementation is close being to fully Verilog-XL compliant. The maturity and the usability of the product is worth a pat on the collective backs of the small team that put it all together. Figure 3. Weighing the average of compile + run times and the memory utilization of the nine designs yielded the performance statistics for the simulators.VCS provides two alternative methods of compiled simulation to Verilog users. The more generic approach is translating Verilog source code into C, which is then compiled and linked using the appropriate C compilers and linkers for a given workstation. The resulting executable is invoked to run the simulation. Another approach is to directly generate object code tuned for a specific architecture and bypass the need for a C compiler. For performance measurement we used native mode, since it gave the best overall performance (compile and run time, and run time memory utilization). The native code approach compiled Verilog models twice as fast as the compiled code approach. This speed appears to have been gained at the expense of memory consumption, however, as it was twice as much as the memory used in the compiled code approach. VCS also includes an incremental compile capability. Incremental compile technology reduces the time to make incremental changes to an existing design because only the portion of the model containing changes needs recompilation. We did not evaluate this feature, other than to verify that it actually works. Unlike Verilog-XL or Silos III, which come with built-in interactive features such as a graphical hierarchy browser, waveform display, and source debug window, VCS bundles a separate interactive user interface and postprocessing environment, Simulation Technologies' VirSim, which makes up for the lack of built-in features. If VCS had a few more of the features of Verilog-XL, such as encryption of ASIC libraries and interfaces to rapid prototyping systems, it would be a more complete and rounded product. Verilog-XL 2.1.2 (Cadence Design Systems, Inc.) Verilog-XL, which dominates the market, is a well-rounded and more complete product for top-down design methodology than any other simulator evaluated here. More than 40 ASIC vendors consider this their "golden" or sign-off simulator. It has over 200 ASIC libraries available to take a design from concept to implementation. We ran Verilog-XL with TwinTurbo mode (+turbo+3, +twin_turbo switches) for performance measurements. We found that the Turbo option ran with negligible penalty during compilation phase over the non-Turbo version (base Verilog-XL). The Turbo version ran two to four times faster than the base product in our performance measurements. As expected, the compliance test cases run with Verilog-XL Turbo (including +turbo+1 and +turbo+2 modes) were fully compliant with the OVISim (Verilog-XL 1.6) and OVI Langauge Reference Manual 1.0. We were, however, able to find some deviations in this compliance when the +turbo+3 switch was used. The extra performance achieved in this mode is realized at the expense of simulation event ordering. CWaves is an interactive interface that provides a graphical environment and is packaged as part of the Verilog-XL Design Environment. It is a waveform viewer that is considerably better than the gr_waves available in previous versions of Verilog-XL. In addition to CWaves, the package also contains a language sensitive editor, a source code debugger, and a design hierarchy browser in both post-processing and interactive modes. Verilog-XL also offers a VHDL model import option that allows you access to VHDL models without leaving the Verilog-XL design environment. VeriBest 3.0.22a (Intergraph Electronics) The PC-version of VeriBest was the winner of last year's PC-based Verilog simulator evaluations. The language compliance, SDF support, and PLI implementation of the workstation version closely rivals that of Verilog-XL and VCS.
Figure 4. The outcome for Verilog simulators was based on the Seva Evaluation Index.
Finsim, from Fintronic USA (Menlo Park, CA), is the simulation engine for VeriBest, which is integrated in Intergraph's ACE Plus design environment. However, it lacks some of the useful interfaces to third-party devices and ASIC Libraries, as well as interactive commands.
Figure 5. The utility of the simulator is measured per $1,000.The performance, both in execution time and memory utilization, leave room for improvement before it can be competitive with VCS and Verilog-XL. Although it uses a compiled code technique (called ACTIVE), the simulation execution time (run time) is about seven times slower than VCS (five times slower when overall performance is calculated). Silos III 94.116 (Simucad) Silos III is an interpreted simulator with the best compilation speed and highly efficient memory usage. In terms of overall performance, it came out third, behind VCS and Verilog-XL. Silos III has a few unique features worth noting. Unlike any of the other simulators we evaluated, it comes bundled with a fault simulator. It also provides a default "trace all nodes" mode. Comparing simulation when all the nodes of a model are traced (using $dumpvars command), we found that the execution and memory efficiency are comparable to those of VCS. Silos III faces a 50 percent time penalty while tracing all nodes (as opposed to no nodes), whereas other simulators face a much larger penalty due to disk I/O. This does not take into account the fact that VCD files (dump files) are taking up considerably more space. Silos III saves all its data in a random-access binary file on disk (save.sim) and makes it accessible through its own waveform display processor. In addition to the waveform viewer, Silos III also has its own graphical interactive command interface and hierarchy browser. In terms of its Verilog language compliance, Silos has room to improve. It does not support hierarchical referencing and has insufficient support for intra-assignment and gate delays. Source-level tracing and single step exist, but both need improvement. Some of the useful commands, such as displaying the current scope or displaying the values of all variables, are not available in the debug mode. The version of Silos III we evaluated did not support PLI. AuSim-VX 1.2 (CAD Artisans) AuSim-VX uses a compiled code approach to create a binary circuit data structure that is loaded into the logic simulator. It also has an interface to Powerview Framework ( Viewlogic Design Systems , Marlboro, MA). In the performance tests, AuSim-VX's compile and link time was comparable to VCS (for those designs it compiled successfully), but the simulation execution time was significantly slower. When a feature/construct is not supported, AuSim-VX crashes, yielding a core dump. It failed to compile four of the nine designs successfully. It comes with good documentation, providing detailed explanation of all its simulator interactive commands with examples, and a sample tutorial. AuSim does not provide a graphical user interface. Its text-entry interface allows tracing of modules/gates across hierarchy by following fan in and fan out. Single stepping is based on time units rather than source lines. With the exception of the "hello.c" test case, AuSim failed to run the PLI tests. VeriWell 2.0B.5a (Wellspring Solution Inc.) VeriWell is similar to Verilog-XL in terms of the supported command line options and interactive commands. The error messages are meaningful and helpful in debugging. Support for the compiler directives and command line options is limited. VeriWell turned out to be the best simulator in our cost-utility analysis; however, for various reasons (some of which appear below) it may not be acceptable to some of the high-end users. VeriWell's language compliance and performance do not compare well with several of the other simulators. In performance tests VeriWell was unable to run five of the nine designs. VeriWell also has no support for PLI, reducing its utility for integration in a design environment that requires interfaces to backplanes, a hardware modeler, or rapid prototyping systems. The documentation is limited and does not include the Verilog Language Reference Manual. The Verilog Hardware Description Language by Phil Moorby and Don Thomas (Kluwer Academic Publishers, 1994), comes bundled with a PC version of this simulator. Viper 2.12 (InterHDL Inc.) Viper has made its command line options, interactive commands, and compiler directives identical to Verilog-XL; however, only a small subset of all Verilog-XL options is currently supported. Viper provides a graphical front-end that supports both interactive simulation and post-simulation debugging. It provides a waveform viewer and RTL-spreadsheet for simulation output at a higher level of abstraction. It also has an interactive interface with System Science's Magellan.
Viper scored low in
performance and language compliance. We could run only two of the nine designs successfully: a small gate level model and a small mixed model. For many of the designs, we saw the message, "internal error, code = 6xx, call support."
integrated system design March 1995[ Articles from Integrated System Design Magazine ] [ ICs and uPs ] [ Custom ICs and Programmable Logic ] [ Vendor Guide ] [ Design and Development Tools ] [ Home ] For more information about isdmag.com e-mail cam@isdmag.com For advertising information e-mail amstjohn@mfi.com Comments on our editorial are welcome. Copyright © 1996 - Integrated System Design Magazine |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Home | About | Editorial Calendar | Feedback | Subscriptions | Newsletter | Media Kit | Contact | Reprints| RSS|
Digital| Mobile |
| Network Websites |
|
International |
|
Network Features |
|
|
|
All materials on this site Copyright © 2009 TechInsights, a Division of United Business Media LLC All rights reserved. Privacy Statement | Terms of Service | About |