PC performance tests-the specialized domain of power junkies and information technologists-reached prime time last week when Advanced Micro Devices Inc. charged that a widely used industry benchmark, SYSmark, had been revised this year to favor arch-rival Intel Corp.
AMD, Sunnyvale, Calif., said that SYSmark 2002 dropped tests from the previous year's version that had showed the Athlon XP 2000+ microprocessor outperforming Intel's 2GHz Pentium by 6% in office applications and 1% in Internet content creation. This year's benchmark, testing the same processors, showed Intel ahead by 4% in office applications and 1% in Internet content, according to Hal Speed, AMD's senior marketing manager.
The SYSmark tests are developed by an industry consortium, BAPCo, Santa Clara, Calif., which includes AMD, Dell Computer, Hewlett-Packard, IBM, Intel, Microsoft, and ARCintuition. Speed said AMD joined BAPCo in July of this year to make sure the SYSmark 2003 benchmark is fair toward Athlon processors.
AMD has its own processor performance scale to compete against higher-clock-rate Intel Pentium 4 MPUs, but also has a vested interest in industry performance tests like SYSmark.
"Benchmarks are critical for AMD in its marketing approach" and the company doesn't want its own performance rating scheme to be negatively impacted by industry benchmark tests like SYSmark, according to Tony Massimini, an analyst at Semico Research Corp., Phoenix.
But AMD's interest in benchmarks is probably much greater than that of the PC buyers at whom the tests are aimed. Analysts said benchmarks have lost much of their allure in recent years, except among some power mavens and technical gurus.
"Most end users are no longer driven by benchmark tests as they were several years ago; price is much more of a factor now," said Bert McComas, principal at InQuest Research Inc., Gilbert, Ariz. The conflicting results of various benchmark tests, McComas added, tend to confuse buyers, who then put less stock in the disparate figures.
Processor performance benchmarks also have little impact on PC makers when they are deciding where to procure MPUs. And as PC buyers become less enchanted with benchmarks, OEMs have even less reason to pay attention to the tests.
An IBM Corp. spokesman in Raleigh, N.C., said, "For most customers, raw [PC] performance is not as critical to the purchase decision as it once was. Increasingly, the decision is being driven by such features as security, manageability, and wireless capabilities."
Kevin Krewell, an analyst at In-Stat/MDR, formerly MicroDesign Resources, Sunnyvale, stressed that benchmark tests' principal value is to serve as a marketing tool.
"It's difficult to have confidence without knowing how the tests are run, specifically what tasks are being tested, and the weighting assigned to each test," Krewell said.
The SYSmark 2002 revised performance benchmarks were the target of AMD's protest.
AMD is providing documentation of SYSmark 2002 application tests whose validity the company questions. For example, SYSmark 2001 tests of Adobe Photoshop showed Athlon beating Intel's Pentium 4 in eight of 13 parameters. AMD's Speed charged that all eight tests were dropped from SysMark 2002. SYSmark 2001 Microsoft Access tests and step-frame flash tests that showed Athlon excelling were also dropped in SysMark 2002, he said.
In one SYSmark 2002 application test, however, Microsoft Powerpoint 2000, AMD increased its lead over Intel from the previous year's benchmark. Of the 14 applications tested, SYSmark 2001 showed AMD leading in eight, one even, and Intel ahead in five. That flip-flopped in SYSmark 2002 tests of the same processors, with Intel leading in eight, one even, and AMD ahead in five.
A BAPCo spokesman said the group is committed to providing the most objective and robust benchmark standard possible. He said he couldn't comment on AMD's criticism of SYSmark 2002 because BAPCo members' deliberations are confidential. "BAPCo is unable to disclose or discuss information provided by each vendor," he said.
Speed said AMD is satisfied with its reception at BAPCo since the company became a member. "We're working in technical committees to help draft the SYSmark 2003 benchmark and have found the process fair."
Speed added that AMD has asked to meet with the BAPCo board to discuss restoring the dropped tests from SYSmark 2002.
"We want the BAPCo benchmark to be a credible source and satisfy the needs of the market," he said, adding that he didn't feel it was too late this year to make the AMD-requested changes to SYSmark 2002.
An Intel spokesman in Santa Clara referred all inquiries to BAPCo. "The Pentium 4 has proven to be the highest-performance desktop on dozens of different benchmarks, not just the BAPCo benchmark," he said.