EEMBC members-which include such industry heavyweights as ARM, AMD, Hitachi, IBM, Motorola, NEC, Philips, STMicroelectronics, Sun and Toshiba-agree that Mips and Dhrystones do not tell what an embedded processor will do in real-world applications like communications, networking, automotive and consumer electronics. Their proposed solution is to develop a number of processor-independent kernels, run them on their machines, present them to EEMBC labs for certification and publish their results-both on the EEMBC Web site and in their own data sheets.

The problem we journalists see is that, after going through the painstaking exercise of perfecting and running those kernels, manufacturers will not, in fact, go on to publish. Last spring, for example, I generated an article that was supposed to represent the first results of the EEMBC benchmarking, scores that would enable users to fairly cross-compare the ARM920T, the Infineon SAB C167CS and the Motorola PowerPC 555. Yet were it not for the carrot-and-stick approach taken by EDN Editor Markus Levy (who serves as EEMBC chairman), these manufacturers would not have released their scores.

In fact, the press relations person for ARM (which scored pretty well on these kernels, even the automotive ones) acted as if the client were a soft-drink manufacturer with a bad batch of cola. The flack refused to comment on the benchmarks until he had compared ARM's scores with those of its competitors. The truth about benchmarks is that everyone is afraid of them. The same manufacturers that otherwise put a lot of money and manpower into the EEMBC effort now live in fear that somebody else is going to look better than they did on one kernel task or another.

Levy, who deserves a medal for his patience, has spent a significant amount of time recently in Detroit, talking up the EEMBC benchmarking with automotive manufacturers. His thinking is that EEMBC members will be encouraged to publish their benchmark scores when end users, like those in the auto industry, demand to see them. Until then, he remains sanguine about the efforts to build an industry-standard benchmark for embedded processors. Still, said Levy, "If you build it, they might not necessarily come."