Servers poised to ride 10G Ethernet

SAN JOSE, Calif. – Server motherboards will sport 10 Gbit/s Ethernet next year, marking the link's long-awaited volume ramp, said engineer and serial entrepreneur Andy Bechtolsheim. Intel Corp. agrees and plans to put its own chip on many of its boards.

"10 Gigbit Ethernet was at a cost premium, but we expect to see that changing in the next year or two as servers incorporate it on the motherboard, and then it will become the lowest cost per port fabric," said Bechtolsheim who co-founded Sun Microsystems and is now chairman of Arista Networks, a maker of 10GE switches.

In a talk at the Supercomputing 2010 conference, Bechtolsheim was bullish on high performance computers hitting exascale performance by 2018. Many of Arista's 600 customers are using supercomputer-class performance for applications such as high-speed financial trading, he said.

The most recent Top 500 list of supercomputers shows Gigabit Ethernet dominates as an interconnect due to its low cost, but it is on the decline. Infiniband which provides faster throughput and lower latency is on the rise.

"Infiniband has been ahead in latency and speed, but Ethernet is finally catching up," said Bechtolsheim.

Infiniband delivers 32 Gbits/s throughput and less than two microseconds of end-to-end latency today, compared to more than 10 microseconds latency for 10GE, said Bechtolsheim. However, use in volume servers is expected to drive 10GE costs down dramatically while Infiniband will remain at a cost premium.

"In cost per performance Infiniband still wins, but in cost per port 10GE will win," he said.

The arrival of relatively low cost, low power 10GE chips for use with copper cables is the gating item driving the link's use in servers, said Steve Schultz, director of marketing for Intel's LAN access division. Indeed, market watcher Dell'Oro Group projects 10GBase-T chips used on sever motherboards will rise from about 5 million in 2011 to more than 25 million in 2014 (see below).

"We are at the beginning of the hockey stick," said Schultz. "We believe the industry is waiting for 10GBase-T," the approach using relatively traditional copper cables, he said.

Intel showed in September its Twinville chip that combines a two-port 10GBase-T physical layer chip from Teranetics (acquired by PLX Technologies) along with Intel's media access controller. The 40nm chip consumes less than 10W, so it does not require a heat sink, and fits into a 25mm² package.

The device marks a huge improvement over Intel first cards in 2007 that consumed 25W per port and cost $1,000. A 2008 card also designed in partnership with Teranetics consumed 15W per port and had only one port.

"We are sampling it now and many OEMs have powered it on server motherboards working in labs," said Schultz. Intel's schedule is to launch it in tandem with its Sandy Bridge server processors in the second half of 2011, he added.

Other companies with 10GE chips include Broadcom, Solarflare Communications and Mellanox Technologies which also is the sole supplier of Infiniband chips.

Managers of big data centers have said they will need Terabit Ethernet switches by 2015 to aggregate traffic from the new servers using 10GE. That has raised alarms with engineers setting Ethernet standard that semiconductor technology cannot keep pace with demand.

Bechtolsheim downplayed the tensions. "I would not say there is an overarching problem here," he said.

"The 40/100G standard is complete and being incorporated in next-gen silicon, so people can purchase off-the-shelf chips soon," said Bechtolsheim whose company relies on merchant Ethernet chips.

"There needs to be a Terabit Ethernet standard, but there is no burning need to do this yesterday," he said. "There may be another step in between 100 and Terabit Ethernet, and it typically takes ten years to deliver a ten-fold performance boost in Ethernet, so I wouldn’t expect to see Tbit Ethernet until 2020," he said.