SANTA CLARA, Calif.—LinkedIn designed its own network switch and is about to switch on its first mega data center. The Web services company also has defined a distributed architecture for its point-of presence centers that could someday use non-x86 processors.
The news revealed in a panel session at the Linley Data Center event here is another example of how large data center operators are defining cloud computing technology. “The days when you needed Arista, Cisco or Juniper to design a switch are gone,” said Yuval Bachar, principal engineer for architecture and strategy at LinkedIn.
Bachar should know. He spent nearly 20 years working at Cisco and Juniper before joining Facebook in 2013 to help design its Wedge top-of-rack switch.
In just four months in his new role at LinkedIn, Bachar helped evaluate and tailor an ODM switch design based on Broadcom’s Tomahawk chip. The top-of-rack switches will link up to 150,000 servers in a data center in Oregon LinkedIn will switch on in March.
“Broadcom revolutionized this business with silicon that’s so integrated when you look inside the box there is almost nothing there but traces between it and the connectors,” he said, noting rivals including Cavium and Mellanox are also delivering highly integrated switch chips.
LinkedIn initially used Cisco gear for everything including servers as well as switches. The biggest challenge with preparing its own switch was tailoring LinkedIn’s networking software to the 1U-sized system, given open source software stacks are not yet ready for large-scale production use, he said.
Bachar’s team evaluated all the white box switches available and chose one, requesting a slight alternation to integrate to its management system. The top-of-rack switch can be software controlled for supporting 25-100 Gbit/s rates and has no oversubscription upstream to the spine network. It is oversubscribed to the server nodes which generally support 10G Ethernet.
Separately, LinkedIn is developing its own operating system. Like other custom elements big data centers are developing, its focus is a mix of simplicity and performance.
Large data centers may use only a handful of the dozens of features in a modern Linux distribution. “We can write that code more simply and tie the software directly and more tightly to our management and provisioning systems,” he said.
Currently LinkedIn operates about six data centers that support up to 20,000 servers each.
Next Page: Looking beyond Intel Xeon processors