"Toad Sprockets," These are all fair questions. I am not employed by any of the companies mentioned in the articles. I did, however, talk to Intel, ARM, Qualcomm, and NVIDIA, and reached out to Samsung and AnTuTu, before even writing the first article.
The data was obtained from as many different sources as possible, including other publications like GSM Arena, which does an extensive job of publishing a wide variety of benchmarks for new platforms. I also worked with some independent sources to verify the numbers. Since no two tests ever result in the same numbers, I averaged the results for my final figures and tried to account for the revisions of each benchmark when possible.
In terms of my background, you are more than welcome to check it out on LinkedIn under TekStrategist. I was at In-Stat for 8 years before NPD decided to close the division. Being in the industry for 30 years, I would hardly call In-Stat a failure, but it was subject to the politics of two large entities, Reed-Elsevierr and NPD, and I will leave it at that. After In-Stat was closed, I struct out on my own as TIRIAS Research, where I have continued to work with various clients and publications, including EE Times. Note, however, that EE Times is not paying me for my contributions.
Prior to being an analyst, I worked in a wide variety of roles in the semiconductor and embedded systems companies, including Intel, General Dynamics, Motorola, ON Semiconductor, and STMicroelectronics. In total, I have over 25 years of engineering and business experience in the electronics industry. I have done everything from launching rockets to launching multi-billion dollar companies. So, my perspective is much more than that of a reporter, analyst, or blogger that has not worked in the industry. Now you be the judge. Am I a credible source?
And finally, I would argue that what I say and what is said about me does have an impact on my credibility. I have already establish a reputation in the industry for being direct an unbiased, and I have no plans on deviating from my morals for self promotion.
Jim, I am curious, where did you get the test results you mention in your article? Also, is Qualcomm paying you for this blog? I think you have an ethical obligation to be honest about that. Reading all these comments I get the impression that the only people commenting here are employed by either Qualcomm or Intel. :)
I also understand you were formerly a member of the now failed research firm INSTAT. How long have you had your own shingle hanging? Are you really credible? All this geek fuss must be a big boost for you, eh? Doesn't matter what they are saying about you, so long as they are talking about you, eh?
I completely disagree. Many of the "professional" sites just repeat the same story without any analysis at all. That's why it was so easy for Intel, ABI research etc to spread their false marketing. But Jim has done some actual analysis and debunked the claims that ultimately led to AnTuTu fixing their benchmark. That's very rare nowadays, even AnandTech often posts literally what Intel says. And as Jim noted in his other article, are all these sites going to retract their articles now that the claims have been proven false? Any site that hasn't isn't professional.
"Also, it is well known that kernel code in x86 architecture can be tuned to execute code faster because it's CISC architecture and optimized code caching instructions, whereas ARM can't (just ask any Linux kernel enthusiast and will confirm this or check in google: this is what I found: http://www.linuxjournal.com/article/7269?page=0,1)"
This is not a RISC vs CISC debate. However in general ARM RISC instructions do as much work as x86 CISC instructions and are smaller as well. So in that sense x86 doesn't have an advantage.
Now it is true that Linux and GCC are more tuned to perform well on x86 rather than ARM, but this is due to x86 ports existing for far longer and having more people work on them. A few years ago GCC was a very bad ARM compiler, but it's much better today.
No idea what that 8-year old link is supossed to show, we all know how to use GCC, but since AnTuTu compiles their benchmarks themselves, it is essential they use the same compiler and options to get a fair comparison across different architectures.
In terms of my comment about technology vendors always wanting the show their products in the best possible light, I speak from experience. I have worked with and for most of the major players throughout the electronics ecosystem and quite honestly, I don't blame them. It makes smart business sense. However, not pointing out issues can also backfire.
Please see my follow-up article on the topic at http://www.eetimes.com/author.asp?section_id=36&doc_id=1318894&. AnTuTu has revised its benchmark and it had a very negative impact on the scores of the Intel Atom processors. I only cited the K900 test results, but tests on RAZR i were very similar.
If even the benchmarking company admits a problem, then there IS a problem. In any case, this was a great discussion about benchmarks in general.
Jim McGregor (tekstrategist on twitter and LinkedIn)