John Cooley's report on the Verification market segment is valuable in spite of not adhering to scientific methods.
There is no doubt that John Cooley's DeepChip website (http://www.deepchip.com/posts/dvcon07.html) offers valuable services to the EDA community. Even John could not stay in business this long by selling snake oil. I, like most other professionals that observe our industry, pay attention to what he says, debate with him, and value his input. So I was very interested in the results of his census on verification methods and tools.
But I hit a problem at the beginning where, through a questions and answers format, John frames the results. I liked his use of the word "mindshare" but I strongly disagree with his characterization of the results as "scientific". I believe that John used this term to defend the value of the results from possible detractors, but unfortunately it has opened him to potentially more criticism.
John attempts to validate the worth of the results with prose that is a veiled attack on the marketing tactics of vendors and writes:
" Q: Is this a scientific survey?
A: Don't be fooled by this question. Cadence/Synopsys/Mentor/Magma all use
internal surveys with only 90 to 150 respondents to make major business
decisions. (Many times all the EDA vendors really care about is what
their Big Money customers say.) Claiming this Verification Census with
its 818 responses is "not scientific" is just EDA vendor FUD designed to
discredit some part of the census their current marketing disagrees with."
Before anyone identifies me as one of those potential critic I want to state unequivocally that I found the results valuable, as you will see later on. But I must point out that actuaries and statisticians would not agree with John that this is a scientific exercise. In fact, John himself points to a bias in the sample base when he states: "This census is very heavily skewed towards ASIC and SoC designers plus their associated verification engineers."
In addition only 818 of the 25000 persons polled responded. This is a return rate of a little below 3.3%. This is too small a sample to assure a statistically correct analysis of the returns. In addition there are many answers from "anonymous engineers" and thus John cannot state unequivocally that all 818 respondents are in fact either design or verification engineers and not individuals in other functional categories who may provide self-serving answers to his survey.
But the results, even if not obtained following strict polling guidelines, are non-the-less very valuable as long as one does not take them literally and uses them as the basis for arriving at considerate conclusions.
Let me give you an example of how I used the results in a way I found valuable.
Item 3 of the survey measures "mindshare" of simulation tools. One of the featured tools is Modelsim from Mentor Graphics. John points out that while ModelSim had a "mindshare" quotient of 41% in 2004 the values were 35% in 2005 and 35.3% in 2007.
What conclusions can I reach from this data? Of course I must look at the total results for this item in order to frame my conclusions. What the results confirm is that Synopsys in the last three years has paid more attention to the simulation market than in the past. After all it was Synopsys that introduced SystemC and it was the same company that provided the vast majority of the technology for SystemVerilog. But Mentor, contrary to Cadence, has been very effective in updating and modifying ModelSim to address the needs of the market and thus has maintained its share of the engineers' minds.
I also keep in mind John's disclaimer about the profile of his respondents and observe that ModelSim is a very popular simulator within the FPGA users market segment. Thus the total installed market for ModelSim is larger than the 35% share in the result. And finally I do not associate any negative connotation to the "legacy" label, as some might tend to do. To me legacy IP or legacy tools continue to be used because of their intrinsic value, not just for convenience or laziness. So responses that state that ModelSim is being used for legacy reasons do not diminish the power and usefulness of the tool, in fact they augment it in my mind.
So, if a superficial reading of the results may influence one to think that Mentor Graphics is not properly improving or marketing ModelSim, once you use the data properly you can arrive at a totally different conclusion. This is what makes me declare that the results of John's census are valuable. Not whether the polling methods are scientific, but what the resulting data has to say to you about the industry. Analysts, editors, and pundits can only take you so far toward informed positions: the last steps are up to you.