Benchmarking is a tricky business. There are benchmarks and there are benchmarks. I'm sure I'm not alone in my weariness with reading reports on yet another batch of benchmark results.
China-based AnTuTu Labs, for example, is famous for a benchmarking tool they developed for Android Mobile Phone/Pad. "They reverse engineer a phone and they do their own benchmarking, but what's in it [their benchmarking] is not always clear," observed Bier. The best defense for independent benchmarking is transparency, he concluded.
BDTI also made clear that it has no intention of creating a scorecard by adding up the benchmarking results of separate components -- GPU, CPU and others -- to measure the performance of a smartphone. "Instead of adding up results of silly little tests, we are focused on examining a phone based on how people actually use [it]," said Bier.
Beyond constant worries about battery life and network performance, consumers want to know how their smartphones function when streaming audio and video; tweets; checking emails; taking pictures, etc. The issue is not only the performance of each application, but also the responsiveness of these apps, and how fast they can be switched.
Bier said that BDTI is currently developing a suite of 50 tests divided into three categories: network performance (including cellular and WiFi); battery life; and user-related media.
BDTI is developing the benchmarking suite in California, while contracting an undisclosed company in China for testing. Unlike Consumer Reports' testing of smartphones, focused on handsets available to US consumers, BDTI's benchmarking is designed for the global market, including China -- where a vast number of new mobile phones are developed, and used by local consumers for their own new apps.
BDTI's press release quoted Thirty Sun, vice president of Tencent, China's provider of services including social networks, mobile chat, web portals, e-commerce, and multiplayer online games, saying:
China has become the world's most important market for mobile smart devices. As one of China's largest Internet media platform providers, Tencent has insights into Chinese consumer preferences and smart phone use patterns.
Initial public device results of BDTI's smartphone benchmarking are scheduled to be released in late 2013. Details of the process will be available in early 2014. Asked about the fee for obtaining the benchmarking source code, Bier said it hasn't been set.