Benchmark Feedback Readers speak their mind on the EDA platform benchmarks and related issues.

Benchmarking on networks?

To the Editor:
Isn't it possible to benchmark using a networked setup?

It's our experience that copying files between our Suns is much faster than copying files between our PCs. Since a design environment can't possibly place all the files on local disk, that would be a significant parameter for us to consider.

Scott Evans
Principal Technical Staff
Neoparadigm Labs, Inc.
San Jose

James Lee replies:
Our goal in these benchmarks was to show the performance of the platforms, not of the mixed 10/100 network we use at Seva. In general, on these high-end PCs with 100-Mbit/s network cards, file transfer occurs quite quickly.

One way to get high performance is to maintain all your source code on a fast machine and then copy it to the machine on which you intend to run the simulation. That way, for each of your tool runs, the files are local. To clarify, if you use a source code control mechanism, you "check out for read" all of the files just before your simulation of synthesis runs.

Again, the goals of these tests were to benchmark the raw machines in their ideal state. As you can see from the results, the I/O performance was the main differentiator. Storing the applications or data sets over the network would have slowed the machines equally; the tests would then have been prone to error because of varying traffic on our network.

Publish more details

To the Editor:
I found the article "EDA Platform Benchmark: Synthesis" [July] very interesting--and surprising enough that I find it hard to believe. I therefore wished to run some of the tests myself; however, a great deal of objective information is missing from the article:

First, did the Solaris and NT versions of Design Compiler in fact produce the same netlist? If not, the comparison of the two is completely invalid. Likewise, were the netlists compared among the NT machines? I'd be surprised if they were different, but since I don't offhand know how Synopsys seeds the mapping and optimization process, I wouldn't be surprised if they differed.

Second, what versions of the Synopsys tools were used on the Sun? I have found wildly different run times and results among different versions of Design Compiler.

Third, are the actual run time data available anywhere? All of the charts gave well-processed data, with no actual times showing up anywhere in the article.

The earlier simulation benchmark had more in terms of hard data, though some corners were still cut. I think it would be great if your Web site provided a benchmark supplement that included detailed procedures followed for running tests and for collecting and analyzing data, and raw data, including log files.

That would add up to a much more scientific study than the benchmark articles you'd find in a mainstream computing magazine. I do think it's important, though, since what is being examined is a very complex process that generally represents a considerable investment of time and capital.

Thank you for such a thought- and action-provoking article!

Kenneth Ryan
Principal engineer
Orbital Sciences Corp./Fairchild Defense
Germantown, Md.

James Lee replies:
We didn't "diff" the netlist produced, but we did look at the timing and area reports to confirm that the designs were similar. Since many factors can affect synthesis, we felt that similar timing and area were good enough indications of similar results.

Second, we used the 1998.08 version of Design Compiler on both Solaris and NT.

Lastly, of course we do have the actual run times, but presenting our results as relative percentages shows more clearly the performance of the platforms: this is a platform benchmark, using synthesis as the comparison tool. If you benchmark hardware with specmarks or winstones, they might not show the performance you'd see when using an actual application; that's the reason we're benchmarking the platform using EDA applications. ISD has contracted with Seva Technologies to perform the benchmarks because we have experts in ASIC design that use these tools every day, enabling us to select a well-rounded set of tests that don't favor any hardware in particular.

Since this was a platform benchmark, and not a synthesis benchmark, we don't want you to use the absolute times of these runs to compare different versions of Design Compiler or other synthesis tools. I would encourage you to establish your own baselines for any benchmarks you want to do.

Regarding your request for data, we've provided the input files, so you can reproduce the results. The article explained our measurement methods. The source data includes a set of batch files and shell scripts that can be used to reproduce the tests.

We don't care?

To the Editor:
I nearly choked reading Robert Baden's statement that we designers don't care what operating system we run on ["Windows NT: Adding to the EDA Arsenal," July, in "EDA Platform Benchmark: Synthesis"]. Of course we do! I want to run my job on a trustworthy machine that I know won't crash when I kick my job off at 1:30 in the morning with a deadline only a few hours or a day away. I then want to go home and be able to sleep soundly, knowing the job will run and I'll be able to come in and analyze the results the following morning.

ASIC design is all about tight deadlines, and that necessitates having reliable tools that work. NT hasn't reached that juncture yet, but Linux has--it has proved itself time and time again as a stable platform. As an ASIC design engineer, I'm expected to produce products that work as specified. No one in his or her right mind would release a chip design that was only partially operational, that crashed every few days when it got a bit hot or cold. As an engineer I can respect the workmanship and reliability of Linux. NT, on the other hand, is still getting there.

Returning to the sidebar, I see that Robert is a Synopsys "head," a manager in charge of new product development no less! I think that if this comment accurately reflects his attitude and his current view of the EDA marketplace, I won't hold my breath waiting for Synopsys to port any of its tools to Linux.

Padraig O'Mahony
Design engineer
Silicon & Software Systems, Inc.
Dublin
(on contract to Philips Semiconductors, Sunnyvale, Calif.)

Robert Baden replies:
I wholeheartedly agree with Padraig's central point: "ASIC design is all about tight deadlines, and that necessitates having reliable tools that work." With regard to Windows NT, beta test customers have told us that they're now ready to use the NT version of Design Compiler in a production environment. These companies have actually used the tool to design real chips in an NT environment. They've seen for themselves that they can reliably produce products that work as specified.

Speaking up for Solaris

To the Editor:
I question the applicability of your benchmark suites when a PC with Windows NT--notorious for requiring reboots with each piece of software added or removed--touts (in your tests) 66 percent faster performance over a Sun Ultra 60 ["EDA Platform Benchmark: Synthesis," July]. I guess you're not testing what's required of each OS when adding or removing software. In the real world, software must be added and often removed from server systems that must remain running while that is done. My experience with NT (and Microsoft products in general) is that rebooting is a "necessary evil" that can occur at any time, not just when adding or removing software. If you test only the number of reboots required for a system to stay up and stable during a week or month (or longer), you'll find that the Solaris system will stay up far longer than the NT system. Now that's a benchmark that's worth something!

Solaris is, has been, and will be the answer to stability and performance now and in the future.

Bill Norton
Course developer
Sun Educational Services
Sun Microsystems, Inc.
Mountain View, Calif.

James Lee replies:

Regarding the applicability of the benchmarks, your points echo comments made by many readers. The benchmarks are quite applicable, since we're testing specific EDA applications in typical use models. For this issue we reran the Verilog benchmark suite on the Ultra 60 and on the 400-MHz PCs ["EDA Platform Benchmark: Simulation II,"], and you can see that the machines perform differently for Verilog-XL and Design Compiler. Benchmarking a machine with the target application is the most appropriate way to compare platforms. Benchmarking reboot time, software installation time, and frequency of reboots are interesting, but we hope that EDA users spend most of their time running applications not installing them. In another article in this issue ["Linux Has What It Takes for EDA,"], I address some of the perceived instability and installation issues with NT.

Finally, I look forward to Sun participating actively in these benchmarks. When faster Suns are available, is Sun willing to participate in or sponsor the benchmarks? Seva Technologies works primarily on Unix (Solaris and HP-UX), and we were surprised at the results as well. We have no hidden agenda on these tests and are willing to test any application on any platform for anyone who wants to sponsor a test. As contractors to Integrated System Design, we provide an unbiased expert test.

To voice an opinion on this or any Integrated System Design article, please email your message to miker@isdmag.com.

integrated system design  September 1998