Group to define benchmark for deep packet inspection

SAN JOSE, Calif. -- The Embedded Microprocessor Benchmarking Consortium (EMBC) is developing a benchmark for measuring the performance of deep packet inspection on network processors. The group already has a draft in hand and hopes to have a specification completed by the end of the year.

A group of about a dozen network processor vendors and communications systems makers have been working on the so-called DPIbench for some time. The metric likely will report a system's throughput in bits per second while checking for a range of viruses and malware programs.

Embedded processors for communications systems used to just read a few bits on the headers of packets, such as its source and destination addresses. But the latest chips scan nearly every bit in a packet to determine the nature of its content, a job that can slow network throughput depending on how it is implemented in silicon and systems.

"The problem is today end users don't know what performance they will get," said Jeff Caldwell, R&D director at SonicWall and chairman of the DPIbench working group. "The DPIbench should provide the real numbers they can recognize on their networks," he said.

The effort is taking the SPECmark benchmark as a role model. DPIbench will not try to report the how well network processors stop a range of viruses and malware programs, just how much network throughput they deliver while checking packets.

The working group includes representatives from about a dozen companies including Cavium Networks, Intel, LSI and NetLogic as well as about four communications systems companies, most of which are choosing to remain anonymous so far.

"We would be very supportive of any benchmark that does a better job showing real world performance," said Ron Jankov, chief executive of NetLogic. "These days many vendors just report peak performance based on their data rates times the number of cores on their chips," he said.

Ultimately, the industry needs a range of benchmarks to handle differences such as the types of cores used and whether they support multithreading, said Joe Byrne, a senior analyst with market watcher the Linley Group (Mountain View, Calif.). But initially, "anything is a step forward bed because we do have a lack of clarity," said Byrne.

Indeed, Cavium, NetLogic and Freescale are battling it out with processors that use very different kinds of cores and multicore implementations. The benchmark "is one of the more challenging things we have done in EEMC because it involves the processors, systems and even software stacks for them," said Markus Levy, president of EEMBC.

The working group is seeking the widest possible review of its draft and is still open to new members. "We want to make sure everyone can work through their issues," said Caldwell.

The final spec is likely to have several different levels of certification. They will cover systems ranging from home routers and gateways scanning for 512 virus signatures to central office systems that check for as many as 100,000 signatures.

The group is likely to recommend changing on a regular basis exactly which group of viruses and malware programs are embedded into test data flows when running the benchmark. "If we only test for certain viruses, vendors could optimize for them," said Caldwell.

 "One of the controversial issues here is how to balance standard stability with the changing nature of viruses," said Levy.