SAN JOSE, Calif. — A new benchmark suite for scaled-out servers is in the works with the first piece of it expected early next year. The processor-agnostic metrics aim to set standards for measuring today’s data center workloads.
A new cloud and big-data server working group of the Embedded Microprocessor Benchmark Consortium (EEMBC) hopes to deliver a suite of seven benchmarks. It aims to complete before April three of them — memory caching, media serving, and graph analysis.
“Typically when we go to a server customer they ask for SpecInt numbers, that’s been the traditional benchmarks for servers for a long time, but SpecInt is not a very good metric for distributed data loads or available instruction and memory parallelism,” said Bryan Chin, a distinguished engineer from Cavium.
Chin’s colleague, Narayan Iyengar, a lead software engineer at Cavium chairs the group which includes about seven members so far including ARM, Intel and Imagination Technologies. Several other companies including AMD and server OEMs have been invited to join.
The group hopes its benchmarks can be run on clusters of less than ten systems in a lab and be accurately projected to forecast performance on thousands of machines typically used in today’s data centers.
One of the big challenges in delivering a processor-agnostic benchmark is the relative maturity of the tools. “Java JIT compilers are just coming out for ARMv8 SoCs, so they are very new, but the ones for the x86 are very mature,” said Chin, whose company is developing its first family of ARM server SoCs.
After the first suite of benchmarks the group aims to target four other areas with no specific deadline yet on when it can deliver benchmarks for them. They aim to measure unstructured data operations, web search, Hadoop/Map Reduce workloads such as web indexing and web front-end serving.
The initial suite of three benchmarks include one that will define a specific configuration for the popular memcached application. Another will set standards for measuring media serving, setting specific levels for acceptable latency and bit error rates.
The third benchmark will measure graph analytics — the speed at which systems such as the Google page-rank algorithm can determine relationships between data. “We’ve identified some workloads we want to run, so we need to pick one graph analytics package and make a standard configuration for it,” Chin said.
Other big-data benchmarks exist such as Cloudsuite, developed by academics in France. “But none are standards backed by a consortium to make sure you get verifiable answers comparable across companies using similar tool chains,” Chin said.
“The existing benchmarks don’t offer a complete view on cloud computing that this benchmark hopes to provide,” said Kevin Krewell, senior analyst with Tirias Research.
“We think it’s a good start and hope more companies make a public commitment to it,” Krewell said. “As more companies get involved the project, the benchmark can develop faster and increase industry traction,” he added.
— Rick Merritt, Silicon Valley Bureau Chief, EE Times 
Related posts:
This is not suprising. Cavium's smaller "wimpy" cores are unlikely to get high spec numbers. EEMBC should make sure that these new benchmarks do not become too specific to a specific architecure or it will not see widespread adoption.
Are there any other silicon vendors participating?