Breaking News
Blog

Benchmarks for Whole Phones Needed

NO RATINGS
Page 1 / 2 Next >
View Comments: Threaded | Newest First | Oldest First
junko.yoshida
User Rank
Blogger
System-level benchmarking
junko.yoshida   8/19/2013 3:09:32 PM
NO RATINGS
One of the most frustrating things for reporters is this. When we ask chip vendors about the performance of their apps processors for smartphones, for example, their answers tend to be specific numbers based on the benchmarking results of CPU and GPu; or they would respond: "Well, it all depends on use-case scenarios."

I'd love to have some answers based on system-level benchmarks -- a tool that measures the performance of a whole phone, based on specifics that consumers care about.  

rick merritt
User Rank
Blogger
Re: System-level benchmarking
rick merritt   8/19/2013 7:36:56 PM
NO RATINGS
So I guess the door is open for someone to go to Mediatek, Nvidia or Broadcom and develop a competing smartphone benchmark.

 

DMcCunney
User Rank
CEO
Re: System-level benchmarking
DMcCunney   8/20/2013 1:46:17 PM
NO RATINGS
Structured in a manner that will emphasize the strengths of their offerings?

It reminds me of the comment "The nice thing about standards is there's so many of them!"

I don't have a problem with BDTI being funded by Qualcomm to do this, as long as they are open about their methodology, what they are measuring, and what conclusions might be drawn from it.  Someone has to pay the costs of developing such a thing,

Once they have something they think works, the next step is to pitch it to the appropriate industry consortium as the standard way you measure what the benchmarks track.  Then you make popcorn and sit back and watch the fun.

 

Max The Magnificent
User Rank
Blogger
Re: System-level benchmarking
Max The Magnificent   8/20/2013 1:55:12 PM
NO RATINGS
@DMcCunney: It reminds me of the comment "The nice thing about standards is there's so many of them!"

Another good one is "Standards are great .... everyone should have one!" :-)

DMcCunney
User Rank
CEO
Re: System-level benchmarking
DMcCunney   8/20/2013 3:30:56 PM
NO RATINGS
And you wind up with a paraphrase of the line about opinions: "Standards are like assholes.  Everybody has one."  With the corollary "incompatible with everyone else's..."

junko.yoshida
User Rank
Blogger
Re: System-level benchmarking
junko.yoshida   8/20/2013 3:26:25 PM
NO RATINGS
DMcCunney, I agree. As long as BDTI makes its benchmarking methodology transparent, smart people in the industry should be able to spot if there are any anomaly, convenient omissions or modifications done to the testing itself.

DMcCunney
User Rank
CEO
Re: System-level benchmarking
DMcCunney   8/20/2013 3:34:40 PM
NO RATINGS
Ideally, I want something that is effectively open source: I want to be able to reproduce the test environment, run the same tests on the same gear, and get the same test results as the people posting thier benchmarks.  If I don't, something is off somewhere.

Patk0317
User Rank
CEO
Re: System-level benchmarking
Patk0317   8/19/2013 9:20:38 PM
NO RATINGS
Junko,

Benchmarks should be renamed to Benchmarketing. (Did I just coin a new term?). I have been involved with becnhmarks since the PREP benchmarking days for FPGAs in the late 1980s and the early 1990s which were gamesmanship at best. Benchmarketing - that's what it is - your mileage may vary ...

Patrick

junko.yoshida
User Rank
Blogger
Re: System-level benchmarking
junko.yoshida   8/19/2013 10:22:39 PM
NO RATINGS
Patrick, "benchmarketing" ain't bad.

The reality is, though, no matter how skeptical and cynical you might get with benchmarketing, benchmark results are often touted and marketed aggressively by those who have money, power and vested interest. 

Susan Rambo
User Rank
Blogger
Re: System-level benchmarking
Susan Rambo   8/20/2013 3:46:14 AM
NO RATINGS
Like the "benchmarketing" term. (I noticed BDTI is marketing its benchmarket study in this article--marketing benchmarketing.) It will be interesting to see the results of the study. I'd like to compare BDTI results to Consumer Reports'.

Susan Rambo
User Rank
Blogger
Re: System-level benchmarking
Susan Rambo   8/20/2013 3:42:11 AM
NO RATINGS
I think testing should happen in US and China at least during the first year of test. I'd like to see if there's a difference in results. Interesting story. Thanks, Junko.

 

junko.yoshida
User Rank
Blogger
Re: System-level benchmarking
junko.yoshida   8/20/2013 8:35:34 AM
NO RATINGS
Susan, I too think that including Chinese smartphones in the mix is very interesting. Considering the growing number of mobile users and smartphones in China, the move is timely. 

prabhakar_deosthali
User Rank
CEO
Simple benchmark from the users' perspective
prabhakar_deosthali   8/20/2013 9:25:55 AM
NO RATINGS
In my opinion, benchmarking of smart phones as far as the end user is concerned should be done on the following criterion

1. The Battery recharge interval

2.  The no of keys/touches required to perform a function

3. How fast the phone boots up

 

This simple criterion will test the processor speed, memory access, UI and power consumption.

 

junko.yoshida
User Rank
Blogger
Re: Simple benchmark from the users' perspective
junko.yoshida   8/20/2013 10:20:40 AM
NO RATINGS
@prabhakar, I like your #2 on the list. How many key strokes (or touch) are necessary to get to the function you need to use is important.

 

DMcCunney
User Rank
CEO
Re: Simple benchmark from the users' perspective
DMcCunney   8/20/2013 1:51:01 PM
NO RATINGS
I like #2, too.  But the problem is that BDTI is attempting to develop hardware benchmarks, and the number of strokes/touches needed to do something is a UI matter, and will be dependent on the OS, the UI it presents to the user, and the apps the user runs. It will have only an indirect relation to hardware, and a well designed phone might handily beat the competition on that measurement while having the least poswerful hardware.

junko.yoshida
User Rank
Blogger
Re: Simple benchmark from the users' perspective
junko.yoshida   8/20/2013 5:35:32 PM
NO RATINGS
DMcCunney and MP, I think you are both saying the same thing.

Hardware benchmarks can't cover the UI or OS matters. 

Well, maybe it's high time for some bright marketing people to come up with UI benchmarketing!

DMcCunney
User Rank
CEO
Re: Simple benchmark from the users' perspective
DMcCunney   8/20/2013 7:04:47 PM
NO RATINGS
Well, maybe it's high time for some bright marketing people to come up with UI benchmarketing!

That should be hilarious if someone tries, because UIs are so highly subjective.  Your idea of a good UI might drive me screaming from the room.  Which is "best"?

Consider Linux, where you have a variety of desktop managers - Gnome, Kde, Xfce4, Lxde, Unity...each somewhat to very different from the others, and each with a following that insists it is best.  (One reason many folks like Linux is because you have that range of choice in UIs.)

"The least keystrokes/touches needed to accomplish a task" may not be a meaningful benchmark.  What if it only takes one or two to run the app and perform a function, but if you use more you get extra options and better control? (Like when you're taking a picture with your phones camera, and you might get a better result if you don't just take the phone app's presets)  An app that take one click to bring up the photo app and snap a picture might win on simplicity, but a competitor that used more taps might take a better photo.  Who wins?

I'll settle for marketing and reviews that do a decent job of explaining the design assumptions and giving a feel for how you use the device.

junko.yoshida
User Rank
Blogger
Re: Simple benchmark from the users' perspective
junko.yoshida   8/21/2013 9:46:13 AM
NO RATINGS
As usual, DMcCunney, you made your point crystal clear. You are absolutely right to point out that UIs are highly subjective. And individual users' needs and wants are probably really different.

That said, simplicity of a user interface does win a lot of "average" consumers' hearts. Further, that is one thing that also often jumps out when a user tries a new phone.

As an average consumer, I would pay attention to such matters, even if that is not a kind of things benchmarks are good at in quantifying.

 

DMcCunney
User Rank
CEO
Re: Simple benchmark from the users' perspective
DMcCunney   8/21/2013 3:10:49 PM
NO RATINGS
That said, simplicity of a user interface does win a lot of "average" consumers' hearts.

Indeed, and that's been a major factor in Apple's success.  They put a lot of thought into their UI, and users of Apple gear can tap an icon or select a menu choice and largely expect it to do what they think it will do, the way they think it should do it.

Apple also enforces UI guidleines on third-party apps, so Apple's overall approach is "Have it our way."  Apple preserves the quality of the user's experience by placing restrictions on what that experience is.  Since Apple users can generally do what they want, or get an app that does, they don't see this as an imposition.  Folks who want more control over and ability to customize their device won't agree, but aren't likely to buy Apple gear in the first place.

As an average consumer, I would pay attention to such matters

Which is why the tech sites all carry lengthy reviews that delve into usage, to give users an idea of what to expect if they get the device.

even if that is not a kind of things benchmarks are good at in quantifying.


And you touched on the key point.  Benchmarks quantify, and UI quality is the sort of thing that largely can't be quantified.  Benchmark values will affect it, by measuring capacities of the underlying hardware to do what the user wants when they tap something.  You can assume more powerful hardware will be more responsive, and may even make possible actions that can't be performed on a less powerful device.  But this, too, is something I expect a review to cover.

docdivakar
User Rank
Manager
Re: Simple benchmark from the users' perspective
docdivakar   8/20/2013 9:22:56 PM
NO RATINGS
Thanks Junko, now that I read more of @DMcCunney's comments, he is saying the same thing (or more like I am saying the same thing since he preceded me!).

I think the question boils down to this: since an average consumer of smartphones is not all that benchmark savvy, do these metrics really matter when selecting a smartphone?

MP Divakar

junko.yoshida
User Rank
Blogger
Re: Simple benchmark from the users' perspective
junko.yoshida   8/21/2013 9:36:58 AM
NO RATINGS
Hi, MP.  You wrote:

I think the question boils down to this: since an average consumer of smartphones is not all that benchmark savvy, do these metrics really matter when selecting a smartphone? 


Probably not.

Consumer Reports, for example, offers the service of comparing mobile phones. But I am not sure if that's regarded as highly as their reports on cars. Part of the reason is mobile handsets are cheaper (than cars) and they reflect more individual tastes.

But I believe what BDTI is trying to do is not for marketing their benchmark results to consumers directly, but to give the electronics industry an opportunity to look at the perfomance of a whole phone (rather than individual component). 

 

docdivakar
User Rank
Manager
Re: Benchmarks for Whole Phones Needed
docdivakar   8/20/2013 4:35:59 PM
NO RATINGS
Looks like the industry loves benchmarking -I didn't know the hangover from Dhrystone & Whetstone would last this long! I find this whole exercise amusing and largely place this on the bright marketing folks to find that 'differentiation' now that the mobile computing space is getting more competitive.

I think one point lost in this 'system-level' benchmarking is the definition of the 'system' itself! Smart phones are much more than yesterday's pure computing devices. A phone with the best hardware benchmark can end up losing when it lacks the best operating system the user interacts with.

MP Divakar

Mike2525
User Rank
Rookie
How to benchmark a processor
Mike2525   8/21/2013 5:04:41 AM
NO RATINGS
There is a good conversation on a separate but related topic going on over at SemiWiki. It's a follow-on thread to an excellent post by Paul McLellan specifically about benchmarking processors.  http://www.semiwiki.com/forum/content/2675-how-benchmark-processor.html

p_g
User Rank
Rookie
Benchmark Vs user experience with phone
p_g   8/21/2013 7:09:34 AM
NO RATINGS
Which phone is best? Its like which shirt fits everyone... Every user has different expectation from phone. I have Galaxy S4 and if I dial down the network usage it can sustain battery ife for upto 4 days for me.. good enough, beats iPhone. But for other users where they are looking for stay connected iPhone performed better (for one person I know :) ).

 

Benchmarks can just compare particular aspect of phone. Ok graphics is genious in a particular phone, but how many seconds (not even in minutes) in phone's life have we pushed graphics to that need? LTE can achieve 100Mbps speed but how many pllaces we get signal to hit that speed and even if we hit how many need it? Battery life is purely per user's settings.. a phone can vary its batter life from 1x to 7x (or more) depending upon what function and how often we enable it..

 

I dont believe in a benchmark that can clearly tell which phone is better from other.

junko.yoshida
User Rank
Blogger
Re: Benchmark Vs user experience with phone
junko.yoshida   8/21/2013 4:18:23 PM
NO RATINGS
@p_g, elsewhere in this forum, I think what DMcCunney wrote under the thread "Simple benchmark from the users' perspective" is a perfect response to your comment.

Benchmarks quantify, and UI quality is the sort of thing that largely can't be quantified.   

Benchmarks can't tell you which shirt you should wear, but benchmarks might be able to tell you the thread counts of the fabric used in a shirt.

 

_hm
User Rank
CEO
Take help from teenage school children
_hm   8/22/2013 7:53:08 AM
NO RATINGS
One of the most effective aspect of benchmarking - perofrmance and other aspects - is to give these devices to different group of teeenage schoold children. Probebly they are the most prolific user of these devices and hence can test them best. Their feedbacks are more authentic, less prejudiced. If teenage likes and approves mobile device, it will be a good indication of success too.

 

junko.yoshida
User Rank
Blogger
Re: Take help from teenage school children
junko.yoshida   8/22/2013 9:43:50 AM
NO RATINGS
@_hm, it definitely helps to find out their usage patterns and their favorite apps, thus helping designers of benchmarking to understand what needs to be tested.

Flash Poll
Radio
LATEST ARCHIVED BROADCAST
Join our online Radio Show on Friday 11th July starting at 2:00pm Eastern, when EETimes editor of all things fun and interesting, Max Maxfield, and embedded systems expert, Jack Ganssle, will debate as to just what is, and is not, and embedded system.
Like Us on Facebook

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)
EE Times on Twitter
EE Times Twitter Feed
Top Comments of the Week