Looks like the industry loves benchmarking -I didn't know the hangover from Dhrystone & Whetstone would last this long! I find this whole exercise amusing and largely place this on the bright marketing folks to find that 'differentiation' now that the mobile computing space is getting more competitive.
I think one point lost in this 'system-level' benchmarking is the definition of the 'system' itself! Smart phones are much more than yesterday's pure computing devices. A phone with the best hardware benchmark can end up losing when it lacks the best operating system the user interacts with.
Ideally, I want something that is effectively open source: I want to be able to reproduce the test environment, run the same tests on the same gear, and get the same test results as the people posting thier benchmarks. If I don't, something is off somewhere.
DMcCunney, I agree. As long as BDTI makes its benchmarking methodology transparent, smart people in the industry should be able to spot if there are any anomaly, convenient omissions or modifications done to the testing itself.
I like #2, too. But the problem is that BDTI is attempting to develop hardware benchmarks, and the number of strokes/touches needed to do something is a UI matter, and will be dependent on the OS, the UI it presents to the user, and the apps the user runs. It will have only an indirect relation to hardware, and a well designed phone might handily beat the competition on that measurement while having the least poswerful hardware.
Structured in a manner that will emphasize the strengths of their offerings?
It reminds me of the comment "The nice thing about standards is there's so many of them!"
I don't have a problem with BDTI being funded by Qualcomm to do this, as long as they are open about their methodology, what they are measuring, and what conclusions might be drawn from it. Someone has to pay the costs of developing such a thing,
Once they have something they think works, the next step is to pitch it to the appropriate industry consortium as the standard way you measure what the benchmarks track. Then you make popcorn and sit back and watch the fun.