If I were with Google, I wouldn't be satisfied with shaking up SDN, or tweaking out some minor mod of a conventional CPU node. I think that's what's so tedious about all the coverage of FB's boring form-factor changes.
Google is in a position where they can look at fundamental changes in programming mode, in ways that conventional suppliers can't. For instance, GPUs have demonstrated that there's a LOT of parallelism out there, in spite of the horrible programming model. Google could be putting dram in-package. They could find a nice uniform way to address large numbers of these nodes (sort of a merged network/dram fabric). If you really go SDN, it doesn't make a lot of sense to stick with the artifacts of traditional ethernet designs (subnets, vlans, ISO layers).
People usually think of this "custom or not to go custom" question as hinging on how much of the conventional architecture can be jettisoned. (ie, if your nodes have nothing but cpu, dram, flash and fabric, you sure don't need 8 ports of SATA or a 6-port USB3 controller. but you probably do want some kind of management coprocessor) But Google should be thinking about more fundamental change, not just subtractions...
Rick, whether you are Google or Baidu, I can't see any benefit in just doing Vanilla processors that you can get from any ARM or x86 vendor. I'd imagine that you want to integrate CPU, GPU and fabric into a single low latency, highly integrated SoC. I'd imagine that something highly optimized for compute along the lines of what you'd get if you integrate Xeon + nVidea + fabric would be where you want to go to cram in a lot of compute at data center scale with low latency without the overhead of a lot of NICs etc and keep system level power down. Trying squeeze down power and latency around what Intel gives you is probably not going to cut it.
There's no point in Google making consumer chips. Allwinner and MediaTek can make such chips cheaper than Google. It would be like Google making its own phones -- they're much better off setting the software standard and letting their partners compete against each other to drive the price down.
Besides, why does Google have Android? So that people can get access to mobile advertising, because that's how Google makes money -- selling advertising. And they target their advertising based on what you've searched for. Search is the crown jewels, and it takes a phenomenal amount on computing power and electricity.
So here you have a well-defined highly-parallel problem. Sounds like a perfect application for custom silicon. Instead of interpreting search software on a general-purpose CPU, do it directly in hardware so that data isn't being copied redundantly, which is what really consumes the energy. If custom silicon cuts the Google electric bill in half and doubles the performance of each data center, the silicon pays for itself pretty quickly. And I'm just being conservative with that factor of 2.
I've reached out to a couple Google contacts for an interview, but given the secretive posture the search giant historically has taken around its data center technologies, I don't expect any substantive responses any time soon.
That said, I am ready any time for an interview with anyone who knows more about this topic.