Design Automation

The Great ESDA Shootout Looking at tabular results of a benchmark can be misleading-- here's the whole story. By John Cooley

Every few days or so, I get a request from an engineer asking for the inside dirt on some EDA tool he's thinking of buying. Usually, it's not a problem because either I have personal experiences to share or know what others are saying about various tools. The one true blind spot has been in the ESDA industry. That is, other than wildly varying stories from a mish-mash of early tool adopters and what the ESDA marketeers have been claiming, there's been no real hard data on these funky graphical-entry and so-called high-level design tools, like Summit's Visual HDL, i-Logix's ExpressV, Mentor's System Architect, and Speed's SpeedChart. When the people who run HP's Design SuperCon '96 phoned, asking "is there any industry-wide question that might make for an interesting contest at our conference?," I knew doing a public ESDA benchmark would fit the bill nicely. Hence, the Great ESDA Shootout was born.

Designing the contest Vividly remembering how much scrutiny last year's controversial Verilog vs. VHDL contest withstood, I wanted to do my best to create a contest that was a fair comparison of ESDA tools. That is, I didn't want to create a problem that intrinsically knocked out half of the ESDA vendors because their tools couldn't handle, say, datapaths or counters too well. To find out what would be fair for them, I phoned up each ESDA vendor. I told them I couldn't disclose exactly what the problem to solvewould be in this 90-minute benchmark, and asked, "If you had your druthers, what would you definitely want to see and not see in this shootout?"

Very quickly, I arrived at the idea that the contest should consist of hand-coding ASIC designers and ESDA vendors simultaneously working on the same problem. We'd compare how long it took to initially enter a design, how long it took to ECO a design, and finally, what the synthesis quality of the generated/hand-coded Verilog or VHDL was. I'd provide easy-to-use diagnostic testbenchs at each stage in both Verilog and VHDL. Since chip operating speed is the big headache for 99.9 percent of ASIC design in the commercial world, I decided the "winner" of this benchmark would be the one whose design synthesized to the fastest speed. Not wanting to deal with being accused of providing a Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys synthesis script that wasn't optimal for a certain ESDA company's output, I decided to let the ESDA companies put anyone on their design teams (for example, a consulting Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys synthesis expert, if needed) for any part of the contest. For target technology, I chose LSI's 300K because it's a robust library and a good, typical, large CMOS ASIC technology.

All I had to do then was fine-tune the contest design problem by going through an iterative process of calling the ESDA vendors out of the blue to ask specific questions about their tools: Can you handle concurrent state machines? How about asynchronous resets? Mixed Mealy/Moore? Random combinational logic? Up/down counters? Do you generate both Verilog and VHDL? What flavor VHDL? Is it synthesizable? Eventually, I settled on three concurrent state machines with asynchronous resets (see Figure 1 ).

Figure 1. The original design was made of three state-machines.

Fear and loathing Oh brother, little did I know what a hornet's nest I was stirring up with the probing questions. Escalade bailed out, claiming they weren't ported to HPs yet. Upon hearing this, one of Escalade's competitors laughed, saying, "They've been on HPs for 18 months; they're just wimping out." Vista Technologies said they were out of the ESDA business and into the hardware/software co-design business instead. Viewlogic politely declined, and later I heard a rumor it was because most of the people who knew anything about ViewFSM had left the company. Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys wanted me to change the contest problem into some sort of vague behavioral form so their Behavioral Compiler could win the day by cleverly juggling states. (Sorry, this isn't a behavioral synthesis benchmark; it's a graphical entry and simulation benchmark.) I also got to go through three long days of negotiations with Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys to secure the use of Design Compiler. "There won't be any other synthesis tools used in this contest? This isn't a thinly disguised synthesis benchmark is it? We don't allow that you know." At the same time, I was being beaten up by the Synplicity people who wanted to enter their synthesis tool into the contest.

Mentor Graphics put me through the most hell by insisting that I rewrite the VHDL testbench (twice) in an extremely generic form, because they wanted to use their non-standard QuickVHDL. They tried to get me to permit them to use their Autologics II synthesis tool. (Sorry, this is an ESDA benchmark, not a synthesis benchmark.) I even found out they were secretly trying to talk all the other ESDA vendors into boycotting the competition and, to top it off, they dropped out of the contest four days before it was going to happen! Fed up, I called their CEO, Wally Rhines. I told him that it appears that fear and paranoia by his ESDA group were going to make his company conspicuously absent from an industry-wide benchmark. Wally, not pleased, said "That's not the philosophy I'm trying to foster here. Winning or losing a benchmark is completely different than running away from one. You can't be a player if you don't compete. I'll see what I can do." Two days later, Mentor was back into the contest.

Opening nightmares Beyond the crisis of getting 40 workstations up, running, and fully networked together at the Santa Clara Convention center, it was a logistics nightmare getting 14 major EDA products (for example, Cadence's Verilog-XL, interHDL's Verilist, and Model Tech's VHDL) from 9 different vendors installed. Because the AEs from each vendor were the only people who knew anything about installing their own software, it was a mob scene of people loading tapes, tweaking stuff, overwriting ".cshrc" files, and then hashing it out with the other vendors, who later overwrote the ".cshrc" file again! Fierce duels over license servers, environment variables, and disk space constantly broke out. It got pretty scary when 10 minutes before the contest began we still weren't certain if we'd have Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys Design Compiler, critical for this contest, up and running on all the machines. The contest came within two minutes of becoming Synplicity synthesis based!

Also, like a fool, the night before the contest, I thought I'd be a clever boy by doing some last minute changes on the design problem. This meant changes to the test vectors, too. (When done, I egotistically thought, "Self, you just redid four complete test suites in less than two hours. Damn, I'm good!") It was during the opening part of the actual competition when I got to personally relearn how God likes to enforce his "pride goeth before a fall" rule for people who suffer from occasional bouts of big-headedness, like myself. I had just explained the design problem and was showing the contestants the template that they were to put their designs in (so they could run in the testbench), when suddenly, someone chirped-up that the port list for the template and the testbench instantiation of it were very different. (Yikes! I forgot to update the template last night! AAAAAAH!) So, I got to go through 40 highly embarrassing minutes in front of 30 miffed contestants and 80 or so whispering onlookers as I nervously fixed both the Verilog and VHDL testbenches before we could officially start the contest (OK, God, I got the message!). Just to make sure I learned my lesson, midway through the contest, an easily corrected typo was discovered in the design data sheet. (Two states were mistakenly shown to have had the same state assignment "0110," yet in real life one was "0110" and the other was "0111." Easily corrected, but scary!)

Figure 2. The changes in the design are highlighted in red.

Winners and losers After the personally embarrassing start to the contest, the rest went fairly well from my point of view and somewhat mixed from other people's point of view (depending on how well they were doing). We got the normal attrition rate of having three engineers panic and walk out once the contest started. Knowledge Based Systems was the only ESDA vendor that said they were coming, even installed their software, and then failed to have a team at the competition. As it turns out, the guy who installed the software had to present a paper during the competition and the rest of their team was away doing a customer benchmark.

Overall, 13 out of the 17 design teams completed the contest successfully, with 9 of these being hand coders. There were only two VHDL- based hand coders in the contest: Nortel's Greg Ward (who ranked midway overall in the competition) and one of the failed-to-completely-finish people; everyone else used Verilog. One guy, an independent consultant named Bob Painter, struck me as kind of odd because he seemed to be spending an awful lot of time reading a Verilog design article that he had clipped from ASIC & EDA magazine. He also was asking me quite a few questions about "vi" and how to use it. It turned out that he had never used Verilog, "vi," or synthesis before, yet, after 147 minutes (going way beyond the contest's 90 minute time limit), he managed to successfully create functionally correct Verilog with ECOs and synthesized it such that it ranked overall right behind Greg Ward, the sole successful VHDL- based designer! (Afterwards, I dubbed Bob Painter the "Miracle Man" because I was so impressed with his courage to go into a public design competition without knowing the tools and then toughing it out to completion. Whoa!)

Mentor Graphics I was surprised at the beginning of the contest to see not only the Mentor Graphics team but also their CEO, Wally Rhines, standing there to provide "moral support." It's not every day you see the CEO of a 2,000- person company showing up at a highly technical function! And I couldn't help but wonder what it was like to be on the Mentor team nervously trying to create and synthesize a design while your boss' boss' boss' boss' boss is watching you. Though, once the contest got going in earnest, Wally left. Ninety minutes later, when the contest was supposed to have been completed, Wally came back and hung around for the remaining extra 77 minutes it took his team to finish the competition. It was really sad. Overall, Mentor got blown away, coming in absolute dead last in every
category: design entry time, ECO time, and synthesis results. Even "Miracle Man" Bob Painter, the guy who had never used Verilog or "vi" or Design Compiler, readily beat Mentor !

Banana Curves

One of the first lessons that any synthesis student is taught in using Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys is the gate area to design performance banana curve. That is, if you synthesize a design to be small, chances are it'll be fairly slow; if you synthesize a design to be fast, chances are it'll have a large gate count. The figure below illustrates this point. The judging criteria for this contest specified that the winner would be the person who produced Verilog or VHDL that synthesized to the fastest design. After the competition, all of the ESDA vendors made wild claims: if they had had more time in the contest to synthesize their designs, they would have done exponentially better. But, looking at the data with Kurt Baty after the contest, we were surprised to find that all the ESDA-generated code followed one banana curve, while virtually all of the hand crafted Verilog or VHDL design followed another, far better banana curve. Kurt summarized this with, "In my mind, this calls into serious question whether the ESDA generated code could ever synthesize to a faster design. Your starting point on the banana curve has a huge impact on how far you can move on it. I don't think Mentor's slow 4.25nsec design could be synthesized to Hoai Tran's hot 1.77nsec design. It's much easier to move to a better speed going up the flat hand-code banana curve than the steeper, ESDA code banana curve." After fitting the second order polynomial to match each banana curve we saw in the data and then linearly projecting to Hoai Tran's 490-gate hand-coded time of 1.77nsec, we estimated that if ESDA-generated code could be synthesized for the same speed, it would be around 1041 gates. Area (LSI 300k gates) Clock-to-clock Time (ns) Comparison of the speed and area. The banana curves represent the function speed = 1/(A*G2+B*G+C). The coefficients ESDA are A = -6.736 e-7, B = 0.0014399, and C = -0.24272. The coefficients for hand coded design are A = -6.33 e-7, B = 0.0076866, and C = -1.6816.

Post competition interviews yielded that some pilot error in their overall methodology, plus some flaws in System Architect itself, hurt them. Instead of exporting their generated VHDL and then running the testbench independently of System Architect, they reported spending 45 minutes struggling to get the testbench into System Architect. Another pilot error was misinterpreting parts of the specs plus using the pre-ECO test vectors on their post-ECO design, causing all sorts of chaos. Three of the System Architect specific problems were (1) the overhead of having to create context diagrams and user-defined enumerated types; (2) having to find a workaround because "state," a reserved word for them, was used in the contest testbench; and (3) System Architect is designed to handle and manage the massive complexities of large, top-down designs, instead of tiny state machines. (That is, they felt they would have kicked butt if the contest consisted of juggling 100 different state machines with large imported blocks of VHDL and 14 levels of hierarchy--like what we find in real design projects. The problem is, it's hard enough to get people to compete in a 90-minute contest, trying to get them to do it for three weeks is a bit much to ask!)

i-Logix One of the major frustrations I had with running the contest was, as judge, I had to force myself to not stop people from doing obviously stupid things, because they didn't plan for ECOs. When I explained the initial design problem to the contestants, I pointed out the symmetry between the top-half and bottom- half of the 15 -state FSM (see Figure 1 ). The i-Logix team liked this so much they re-implemented it as one 8-state FSM plus one flip-flop to keep track of when they were in the top half or bottom half of the original specified 15-state FSM. This was quick, clever, and very much a mistake for them, because when the ECOs came in (see Figure 2 ), the top half and bottom half were no longer symmetric, meaning they had to redo their whole design! Their VP of engineering, a good friend, ripped into me with full anger, shouting "but you specifically pointed out this symmetry at the beginning of the contest!!! This is unfair!" Meanwhile, one of his applications engineers was simultaneously whispering to me, "Ignore him! He doesn't mean that. It's OK. We're OK. He doesn't mean that."

Later on, when i-Logix came in with the best synthesized results for an ESDA vendor and in second place overall, the i-Logix VP was too happy to be concerned. "OK. So you got us by specifying the design in a tricky way. What counts is that we blew everyone away with our final results after redoing the design three times! We won! We won! We won!" (I chuckled and reminded him how he contracted me two years ago to help him change how i-Logix was writing out its Verilog and VHDL. This was because his customers were howling that i-Logix-generated code was synthesizing to big, fat, slow gate-level designs.)

Summit Design Other than one hand-coder using Synplicity's special language-based editor and one other speed-typing hand-coder using "vi," Summit Design cleaned house in the design entry and ECO part of the design contest (see Table 2 ). I say this because during the design entry part of the contest, they had a core dump because there wasn't enough swap space to simultaneously run their tool and the Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys Design Analyzer, which screwed up their initial design entry time. Their ECO time proudly speaks for itself, giving them a full 50 minutes to run Design Compiler. And that was their downfall.

What happened was that pilot error using Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys managed to transmogrify their nice single piece of source VHDL into three subparts that couldn't be glued back together because of VHDL's strong type checking and the use of the type "buffer" on the subpart's ports. Essentially, they got caught trying to be too clever using Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys , and it bit them, big time. They later admitted that if they had used Verilog instead (which doesn't have strong type checking), they wouldn't have been bitten using Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys . As judge, to get their synthesis results, I took their machine- generated source code and did only a single default Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys compile on it.

Speed Electronic The Speed team approached the design by making three completely separate concurrent state machines in anticipation of upcoming ECOs. This meant a hit on their design entry and ECO time, but they didn't care, because they felt they'd win with better synthesizable final code (which is what I specified as being the most important criteria for this contest).

And VHDL bit them, too! Again, it was the port type "buffer" that got them, because their tool converted it to type "inout," which, consequently, caused their code to choke in the testbenches that were expecting port type "buffer" instead of type "inout!" In retrospect their VP of engineering said "we should have used Verilog because we wouldn't have run into stupid type-checking problems. Also, we're going to fix that automatic 'buffer-to-inout-type' conversion bug the moment we get back to the office."

The other clever approach was that they used Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys 's FSM Compiler. You've got to understand that as the guy who runs the E-mail Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys Users Group, for years the entire customer base has been badmouthing FSM Compiler so much that none of the experienced users use it any more. To most of the customer base, it's seen as one of those bogus features that doesn't really work. Well, to my and Kurt Baty's (who helped in the post-contest analysis) surprise, the winner, Hoai Tran of Symbios Logic, as well as the third place team, Speed Electronics, both used FSM Compiler instead of plain old Design Compiler, which 99 percent of the planet is using. Even Richard Weber of SGI (the guy who was using Synplicity's language-based editor to enter the design so quickly), who had a leisurely 55 minutes to use Design Compiler for a variety of multiple synthesis runs, still only landed fourth place overall. This tells me that perhaps it's time that I and the rest of the Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys user community took another look at FSM Compiler.

Table 2
Specific results for each contestant
Original Design Time (minutes) ECO Design Time (minutes) Gate Level Performance (nsec) Minutes Contestant Minutes Contestant Nsec Contestant 22:14 Rich Weber / SGI 7 Summit Design 1.77 Hoai Tran / Symbios 26:57 Brian Korguth / TI 11 Hoai Tran / Symbios 2.09 i-Logix 32:54 Summit Design 12 Mike Moody / TI 2.31 Speed Electronic 40:45 Greg Ward / Nortel 13 Rich Weber / SGI 2.46 Rich Weber / SGI 50:25 Hoai Tran / Symbios 15 Speed Electronic 2.58 Kurt Baty / ind. 51:51 Kurt Baty / ind 15 Greg Ward / Nortel 2.61 John Gilbert / Moto 52:47 Kam Kittrell / TI 16 Kam Kittrell / TI 2.63 Greg Ward / Nortel 57:00 John Gilbert / Moto 18 John Gilbert / Moto 2.94 Bob Painter / ind. 59:52 Mike Moody / TI 20 Brian Korguth / TI 3.16 Brian Korguth / TI 62:51 i-Logix 22 Bob Painter / ind. 3.23 Summit Design (VHDL) 74:32 Speed Electronics 28 Kurt Baty / ind. 3.35 Mike Moody / TI 147:30 Bob Painter / ind. 35 i-Logix 3.41 Kam Kittrell / TI 167:24 Mentor Graphics 43 Mentor Graphics 3.41 Summit Design (Verilog) 4.25 Mentor Graphics

Only Mentor was wounded The week following the Great ESDA Shootout, EE Times ran an article titled "ESDA Wounded During Shootout." Although I agree with most of the written article, I disagree with its title and what it implies. The results of the contest clearly tell me that ESDA tools are definitely up to snuff for some types of designs. In synthesizable results, two out of the top three contenders were ESDA vendors, i-Logix and Speed Electronic. Summit didn't do well in synthesis because it tried to be too clever and, as a result, got lost. For initial design entry time, Summit and Synplicity's language-based editor ranked in the top three; Speed Electronics and i-Logix got caught up in VHDL traps and misunderstanding the spec, respectively. For ECO time, Summit kicked butt and Speed Electronic did well--even after getting caught on VHDL type checking traps. i-Logix did a complete redesign during that time!

The only ESDA company that got hurt was Mentor Graphics because they took a distant last place in every category--even going against a guy who has never even used Verilog, synthesis, or "vi" before! But they had serious pilot error and a few nasty workarounds to find, like dealing with "state" being a reserved word for them. Also, having your CEO looking over your shoulder during a contest might turn anyone's typing fingers to cigars and their brains to mush.

At the post-contest panel discussion, when polled, 12 engineers said they won't consider the use of ESDA tools for their projects, 8 said they would, and 28 said they're interested in language-based editors like Synplicity's. Also, some hand coders in the audience said this problem was handed to the ESDA vendors on a silver platter in its bubble chart form. They further contended that it was not something you had to do a lot of design thinking on. I disagreed: I wanted to measure ESDA tools performance--not the ability to figure out a tricky state machine from a vague word description. Overall, I think this contest validated that ESDA tools can do as well, if not better, than most hand-coding designers for small state machines. But it's just one data point. My gut tells me it's a completely different story once you start considering larger, real-world projects. *

I'd like to thank Hewlett-Packard for sponsoring this shootout at Design SuperCon '96. I'd like to ask my fellow engineers please to e-mail me what they thought about this competition. (Please don't just say you loved or hated it; rather, say why you loved or hated it.)

John Cooley is the founder of the outlaw E-mail Synopsys .com/isdweb/&lf=isd-sendtolog"> Synopsys Users Group (ESNUG) and president of the Users Society for Electronic Design Automation. He works as an independent ASIC consultant and loves hearing from fellow engineers at jcooley@world.std.com or (508) 429-4357.

integrated system design  April 1996