After my previous articles on benchmarking, I thought it only fitting to address some of the issues that arose from the discussions.
As we have seen, performance benchmarks are far from perfect. One reason lies in the history of technical benchmarks. They arose as a way to evaluate PC processors and computing systems. These systems have rather defined applications and use models, and power was an issue only if it threatened peak performance. Unfortunately, that model does not apply to mobile devices.
In mobile devices, applications change rapidly, because they are not based solely on the device itself. The applications change with the features of the device, the network connectivity and cloud services, and the user environment. Similarly, the use model can vary by any number of dynamics and is often unique to the users at any particular time. As a result, the potential use scenarios are exponentially greater than those for a PC. Unfortunately, there is no effective way to test for all potential use models. The best anyone can do is test for a range of models and applications.
Finally, there is power. Many of the comments on my previous articles focused on power. Some even said that performance is meaningless, and that the only relevant measurement is power. It is correct to say power is important; it translates directly into battery life. However, consumers also care about what they can do with a device, which translates into performance. This highlights the dilemma of mobile device OEMs -- balancing features and performance with power. To be truly effective, mobile device evaluations should consider both performance and power, which together translate into platform efficiency (performance/watt).
Unfortunately, measuring power is equally as challenging as measuring performance. Many of the benchmarks set out to compare the mobile processors in these devices. This assumes that all mobile processors are created equal. Every mobile processor on the market is a complex system-on-chip solution, but no two solutions are the same. In evaluating the mobile processors from Intel, Nvidia, Qualcomm, and Samsung, you have the following comparison on processor cores.
(Source: Tirias Research)
No two processors -- even those using the same architecture -- are using the same processor core configuration. As a result, even the measure of a single core would not be an accurate comparison, because all are designed to execute tasks and applications differently. Incorporating all the other system functions integrated into each processor makes the effort even more challenging. With 4G baseband and WiFi modems integrated into the Qualcomm snapdragon processor, one would need to consider the impact of discrete modems and associated components, such as the filters necessary to avoid signal coexistence issues, to conduct a fair comparison. When all such processor differences are considered, the list of additions and subtractions to perform an accurate comparison can be extensive.
Even if all processor differences are considered, the power reading may not necessarily translate into an accurate battery life figure for the platform because of the impact of other system functions like sensors, the display, and the capacity of the battery.
The point is that power is important, but the only reasonable way to measure it is on the platform level, just as the only effective way to measure performance is on the platform level. Even then, it is important to use multiple data points/benchmark evaluations to overcome the issues we've already noted regarding applications and use models.
With limited methods of effectively evaluating mobile devices, we (industry observers) should present a more complete picture in our evaluations, including performance, power, and efficiency. I wish I could say that the issues associated with benchmarks will get solved, but the rapid pace of mobile technology innovation will continue to make this effort challenging. In addition, as some functions migrate to the cloud, benchmarks will have to evolve further to evaluate the specific functions being performed and the total device-to-cloud solution, because evaluating just a mobile device may become a meaningless effort.
Jim McGregor is the founder and principal analyst at Tirias Research.