Breaking News
News & Analysis

Datacenters Drive 25GE Effort

Ethernet group includes Google, M'soft
7/3/2014 09:40 AM EDT
11 comments
NO RATINGS
1 saves
Page 1 / 2 Next >
More Related Links
View Comments: Oldest First | Newest First | Threaded View
Page 1 / 2   >   >>
TanjB
User Rank
Rookie
a practical goal
TanjB   7/3/2014 7:04:24 PM
NO RATINGS
25Gbps is where the transducers are landing for simple, single fiber capacity.  10Gbps underutilizes the hardware.  40Gbps does not look feasible without 2 or more fibers, it is an implausible step up for transducers any time soon.

In a data center with millions of connectors it makes sense to aim at the optimal feasible point which seems to be 25G for the next few years.  It is a good idea to get all the parties interested in buying and selling at this performance point to agree on compatibility.

 

Nicholas.Lee
User Rank
Rookie
10G,40G,100G v.s. 25G.50G,100G
Nicholas.Lee   7/4/2014 4:12:48 AM
NO RATINGS
Good article.

At first glance it might not seem obvious to readers why you would have a 40G standard and a 50G stsndard as they are so similar in speed, but this all needs to be seen through the lens of the SERDES transeiver rates on the chips.

When a 10G SERDES was the fastest transeiver available then it made perfect sense to use bundles of fibres each of which terminated at a 10G transeiver, and so we ended up with standards based on 1, 4 and 10 fibres aggregating to bandwidths of 10G, 40G and 100G respectively.

Now, chips are avilable with 25G SERDES transeivers, so companies are right to revisit the standards and update them to be based on multiples of 25G. Hence bundles of 1, 2 and 4 fibres aggregating to bandwidths of 25G, 50G and 100G Ethernet respectively.

 

rick merritt
User Rank
Author
What's after 25G serial?
rick merritt   7/4/2014 4:46:18 AM
Thanks Nicholas for cutting to the heart of the issue--the rise of the 25G serial link after so many byears of hard work.

It's easy to predict quite a big wave of products wikl ride this technology.

So what are engineers (serdes experts) turning their energies to next? I know there was a 100G serial workshop sponsored by Ethernet Allaince this month, but methinks that's pretty far future stuff, yes?

markhahn0
User Rank
Rookie
IB?
markhahn0   7/4/2014 6:15:02 PM
NO RATINGS
I always find it strange how the eth world bumbles along, apparently oblivious to the IB world.  In spite of IB being almost sole-sourced by Mellanox, and that company appearing in this effort.

56Gb IB (which is admittedly 4-lane, thus 16Gb/lane) has been around for years, and is pretty much the entry level in the computational datacenter.  And it's copper.  So this whole thing is puzzling on two counts: if the demand is there, why are these eth/optical efforts lagging, and why are they insisting on optical?  power can't possibly be the issue, since we're not talking about high-density applications (even a bundle of 25 Gb coming from a rack is never going to compare to the power dissipated by the *compute* contents of the rack, or even disks.  cable *length* instead?

DrFPGA
User Rank
Blogger
Google and Msoft
DrFPGA   7/5/2014 6:32:57 PM
NO RATINGS
crafting standards for Ethernet. I can't wait to see what 'extras' they load us down with. Maybe corrosion protection on the bottom of the connection...

TanjB
User Rank
Rookie
Re: IB?
TanjB   7/5/2014 11:46:41 PM
NO RATINGS
Mark, IB does not scale to the data center.  It is nice for building a supercomputer in a rack with RDMA and a few dozen number crunchers, but not designed to connect 100,000 or more servers in a data center.  That is where the interest in 25G serdes based links comes into play.  Maximum data in minimum counts of connectors and fibers.

Mellanox is playing with Ethenet using RRoCE to try to get some of the IB benefits on an ethernet fabric, but it is unclear if the PFC mechanism used to replace the tokenized pacing in IB will really work with an interesting number of machines and the short bursts of random sourced traffic which characterize the DC.

interconnect Guy
User Rank
Manager
Re: SRIO at 25Gbit
interconnect Guy   7/6/2014 8:06:22 AM
In the end the winner is Ethernet,  it has the backers, and the Eco-system,   Count the Teir One Vendors and Customers supporting the effort, and its clear.

rick merritt
User Rank
Author
Re: IB?
rick merritt   7/6/2014 10:21:28 AM
NO RATINGS
@TanjB: Aree there any hard numbers on # of cluster connections IB can support vs. Ethernet? and SRIO?

TanjB
User Rank
Rookie
Re: IB?
TanjB   7/7/2014 1:17:56 PM
NO RATINGS
GSMD, not clear what about SRIO is so interesting?  At the physical link level (which is the original article) it seems to use the same interconnects as Ethernet (or as you say, anything else).  The existing SRIO protocols seem to use XON/XOFF pacing which is archaic and not scalable in large flat networks.  Without a better and proven answer to the pacing/congestion protocols claims to be more efficient than Ethernet at data center scale will meet justified skepticism.  I'm not sure why you think Ethernet is slow: a VL2 fabric with commodity switches will connect any to any in a 100k pool of computers with about a half microsecond transit time.  RDMA is problematic at that scale since RDMA is a paired connection with dedicated buffer at each end, not ideal for cloud scale data centers where your services may randomly map to 10s of thousands of servers.  Costs of Ethernet tend to be cheaper than alternatives with less competition.  Failure rates in practice are very low and in a cloud data center redundancy is managed through server redundancy not component redundancy.

For SRIO to be interesting it needs to drive costs further down than existing commodity Ethernet 10/40G ethernet, and somehow offer

For an older but still interesting discussion of VL2 networks see:

http://www.nanog.org/meetings/nanog47/presentations/Tuesday/Maltz_monsoon_N47_Tues.pdf

Several companies build their DC networks that way these days.

 

TanjB
User Rank
Rookie
Re: IB?
TanjB   7/7/2014 1:24:34 PM
NO RATINGS
Rick, IB anecdotally tops out around 50 to 100 servers.  Are you aware of anything bigger?  The protocols seem to revolve around a single shared switch connecting all servers, which is a serious limit to scale.

SRIO is pretty much a point to point link at the moment.  Simple, but there are reasons why protocols for large networks are not that simple.  I'd be fascinated to read any case studies of large networks using it.

Ethernet with VL2 or similar Clos switching networks in practice supports data center clusters with more than 100k servers each with at least a 10 GbE link.  Google is thought to have been like that for a while, Amazon and Microsoft too.  It is the baseline for a modern DC.

Page 1 / 2   >   >>
August Cartoon Caption Winner!
August Cartoon Caption Winner!
"All the King's horses and all the KIng's men gave up on Humpty, so they handed the problem off to Engineering."
5 comments
Top Comments of the Week
Like Us on Facebook

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)
EE Times on Twitter
EE Times Twitter Feed
Flash Poll
Radio
LATEST ARCHIVED BROADCAST
David Patterson, known for his pioneering research that led to RAID, clusters and more, is part of a team at UC Berkeley that recently made its RISC-V processor architecture an open source hardware offering. We talk with Patterson and one of his colleagues behind the effort about the opportunities they see, what new kinds of designs they hope to enable and what it means for today’s commercial processor giants such as Intel, ARM and Imagination Technologies.