In most embedded applications, system developers care about high-level system attributes, such as low cost, long battery life and high throughput. System developers generally don't care how a solution delivers these top-level attributes; they care only about how well it does so. Therefore, benchmarks should be built from the top down, based on application requirements-and not from the bottom up, based on preconceived notions of what sorts of hardware will be used in the application.

In fact, the need for flexibility extends beyond accommodating all of the hardware options. To be truly relevant, a benchmark also must accommodate the full range of implementation techniques used in the application. For example, developers of signal processing applications almost never build an entire application with plain C code. Even the best compilers sometimes produce very inefficient code, and when they do, a skilled programmer often can make vast improvements with modest effort-perhaps by modifying the C code or by replacing portions of it with assembly code. As a result, a benchmarking approach that relies on plain C code is unlikely to produce useful performance data for signal processing applications.

Another problem with using C-or any other programming language, for that matter-is that this approach isn't appropriate for solutions such as FPGAs, which do not run "software" in the traditional sense. Of course, there are many applications where the best approach is to use an FPGA or other solution that doesn't rely on software. Hence, benchmarks should avoid narrowly specifying the implementation methodology to avoid excluding relevant hardware options.

In summary, setting out to create benchmarks for a specific implementation approach-such as using multi-threaded processors or plain C code-is going about things the wrong way.

Instead, benchmarks should model the application requirements and leave the implementation approach (multiprocessing, multithreading, reconfigurable hardware, hardwired solutions, or what have you) to the benchmark implementer-just as actual applications allow for many different implementation approaches.

Benchmark developers who find themselves designing benchmarks to show off particular features of particular kinds of hardware should ask themselves whether the results are really going to be meaningful. And system designers need to understand the design of a benchmark before accepting and using the results that the benchmark produces. If a benchmark assumes a specific hardware feature or a specific implementation methodology, system designers should proceed with caution.

Jeff Bier, president of Berkeley Design Technology Inc. (www.BDTI.com), a consulting firm providing analysis and advice on DSP technology. Kenton Williston of BDTI contributed to this column.