Breaking News
News & Analysis

Startup's Cluster Trumps InfiniBand

2/25/2014 03:38 PM EST
24 comments
More Related Links
View Comments: Oldest First | Newest First | Threaded View
Page 1 / 3   >   >>
rick merritt
User Rank
Blogger
Not your father's PCIe
rick merritt   2/25/2014 5:32:10 PM
Can a non-stadard PCIe system interconnect gain much traction?

GSMD
User Rank
Manager
Re: Not your father's PCIe
GSMD   2/26/2014 1:53:22 AM
PCIe as a fabric and an interconnect is a  lost cause ! I wish efforts to make it one would stop. It is a good interconnect for what it was intended to do, connect master devices to I/O sub-systems.
Of course, its technical deficiencies will not stand in its way to becoming a sucesses since marketing trumps technology any day !

Full disclosure: I am a member of the RapidIO trade association but have no commercial interests since I work for a University. Our SRIO controller is also open source.


Hence I would recommend  SRIO which works and you can buy PCIe HBAs today to do the same. Of course using PCIe increases latency but only Freescale and TI CPUs have SRIO controllers. IB offers similar perf. and latency but is pretty expensive since vendors are few.

And there is no push to make it a CPU to CPU interconnect.

interconnect Guy
User Rank
Manager
Re: Not your father's PCIe
interconnect Guy   2/26/2014 5:58:32 AM
sadly RIO or RapidIO is several signaling generations behind, at 10GB/sec  While others are at 27Gb/Sec  (per pair)  and RapidIO switches are single sourced at IDT.

GSMD
User Rank
Manager
Re: Not your father's PCIe
GSMD   2/26/2014 6:41:03 AM
I humbly suggest that your facts are wrong.

 

1. SRIO 3.0 which is the current standard specifies lane speed at 10 Gbit/sec and 25 Gbit/sec and a max of 16 lanes per port. You should see parts this year. The lower speed  is faster than PCIe and the same speed as 10Ge. SRIO is basically tracking Ethernet SERDES standards and hence is assured of market competitive lane speeds.  the higher speed is competitive with other standards. So I think that takes care of your obselete line speed comment.

the encoding is also 64/67b which makes it more reliable than PCIe or 10Ge on longer PCB tracks.

Higher than 25 G speeds are problematic on PCBs and it is not clear on how the industry is going to proceed. We are going optical beyond 25G for longer lenths but may continue to use electrical up to 50g speeds for ultra short lengths.

 

2. All interconnects other than GigE have sourcing issues. PCIe switches are also available only with IDT and PLX and IDT nearly acquired PLX last year. So I am not sure what your point is since it is a problem for all interconnects, not just SRIO.

I had also pointed out that my insititution, IIT-Madras is releasing a commercial grade SRIO 3.0 IP (10 G first and then 25G) with BSD license. We will also jointly release with Xilinx dev kits using Xilinx 10G/25G SERDES so that users have a ready to use FPGA platform. The open source kit will contain all digital portions including the PCS/PMA components. It will take some time but the non-PHY components are already online (bitbucket.org/casl). I would like to release the SERDES also but that involves coordinating with foundries, so we will use 3rd part SERDES for now.

the kit will also include a complete verification IP, again completely free under a BSD lic. More than 20 man years of effort will go into this effort.

Commercial entities have already started evaluating this IP.

That should take care of any second sourcing concern since no other interconnect will have this wide an availability.

In any case you are wrong about IDT being the only IP source. Xilinx, Altera, Praesum, Mobevel and possibly others provide FPGA and Silicon IP.

 

I rest my case !

interconnect Guy
User Rank
Manager
Re: Not your father's PCIe
interconnect Guy   2/26/2014 7:54:53 AM
 

Per the RapidIO website:

"The 10xN specification, backward compatible with RapidIO Gen1 and Gen2 systems, supports 10.3125 Gbaud per serial lane"

The 25GB story is:


"RapidIO specifications are under development to support 25...."


Other interconnects are shipping 25Gbaud Now, and yes IP has been available for years, but never full cores for RapidIO (just thin Phy layers) complete cores are available for FPGA's for interconnects like PCIe and Ethernet.


You may have access to information about roadmaps for Gen 3 RapidIO parts, but the the reality is Infiniband, PCIe and Ethernet are shipping these speeds, and have been for sometime in volume.


In Niche applications the second sourcing may not be a issue, but in volume it is.


Freescale and TI support both PCIe and Ethernet. (and older sRIO) (Gen2)


Per the RapidIO Product showcase last Freescale product for sRIO was 2008
 

GSMD
User Rank
Manager
Re: Not your father's PCIe
GSMD   2/26/2014 10:03:58 AM
1. Site is misleading. spec as publsished today supports 10 and 25G. Some of the features optional for 10G (mainly error correction related) are mandatory for 25G. Since I am implementing a 25G solution today using Xilinx SERDES, I am pretty certain the specs supports 25G. The specs are free online, take a look. You will see PHY only for 10G since 25G SERDES for Ethernet is not final. Once that is final, SRIO will specify that too. But you can implement 802.3bm prelim if you want to go 4x25G optical now. Which is what I am doing and using zQSFP modules (Intel MXC is the other option)

2. I do not what you are referring to as shipping high speed interconnects.

Ethernet is only 10G per lane now. 40G is 4 x 10. 100G is 10x10 and a proposed 4x25.

SRIO uses the same SERDES technology as Ethernet, so by definition it will track Ethernet in terms of speeds. 

PCIe is only 8G today, The proposed higher speed standard is not ready. How can you claim PCIe is shipping at speeds greater than 8G ?

Only Infiniband in the interconnect space is faster. Not incuding FC since it is irrelevant in thsi space).

Among other responsibilities, I am part of the official  interconnect standards effort in India, so you can be rest assured I track these on a  daily basis ! I also used to sell PCIe, SRIO and HT IP for a decade.

To sum up

Ethernet is currently only spec'ed at 10G per lane

PCIe is spec'ed only at 8G per lane

Infiniband EDR (the only finalized variant of Infiniband) is spec'ed at 25G, same as SRIO. HDR IB  will ship only in 2017

If you think otherwise, please show me shipping Ethernet and PCIe parts that have  speeds greater than 10G per lane and IB at more than 25 G per lane.

So of the lot only SRIO and IB are spec'ed at 25G. Granted SRIO is tracking IB by one year but that hardly makes it an antique interconnect.

By the way, there is nothing in the SRIO standard that limits it to 25G. The changes will come mainly in the PHY since error correction becomes a major issue. As you can see from the spec the encoding is conservatively spec'ed at 67/64 since 10Ge had problems with 66/64. Interlaken is similarly conservative.

rick merritt
User Rank
Blogger
Re: Not your father's PCIe
rick merritt   2/26/2014 10:44:35 AM
@GSMD: Thanks for sharing your expertise.

As I underdstand it, A3Cube is claiming advances in latency, not necessarily throughput. Their goal is to enable a single system image over a variant of PCIe.

What do you think about their latency claims?

GSMD
User Rank
Manager
Re: Not your father's PCIe
GSMD   2/26/2014 12:04:06 PM
I had actually dsecribed the latency figures in detail in another post. But bottom line is that SRIO's 100ns latency is the best today. Nothing magical in that, keep the protocol simple, latency would be low. The KISS principle is applied well in SRIO.

PCIe is slightly worse. Having impemented  both IPs, there frankly is not that much theoritical latency difference but SRIO being lighter will have less latency. Most of the latency actually gets swallowed in the PHY. Having said that  switch latencies for PCIe are higher than I would have thought. Just see the public datasheets of PCIe switches from IDT and PLX and SRIO from IDT. I am not making it up.

Let me go thru the latencies of our published IP. 1 cycle for the logical and transport layer. That works out to .5 ns at 2 Ghz. Maybe 1 Ghz is more typical, so 1 ns. The rest is Digital + Analog PHY.

Public SERDES figures are in the 15 ns range. the PCS/PMA layer seems sub 5ns but we have not finished coding yet. CRC itself seems to be 3 cycles. So a 20-30 ns is a good target to aim for if attached directly to the bus. PCIe will be higher.

It is amazing that there is not a single detailed technical analysis at this level comparing the following PCIe, Ethernet, SRIO, IB, FC, Interlaken, QPI.If there was, we would not be having these discussions. Freescale by the way bought the SRIO IP from my prev company and Cray got HT from us too. So I have had to do these analyses for a while now. I do wish these discussions would come to this level so that we can sort out issues at a technical level instead of having to rely on marketing FUD from vendors and trade bodies.

I basically got pulled into all this for two reasons, I occasionally teach Comp. Arch at a master's level at one of India's premier tech universities and i had to select a standard interconnect for the India processor project and the supercomputer project. Settled on PCIe first but it simply did not cut it technically. So after 6 months decided to switch to SRIO.

After all the analysis what I realized was that all standards pretty much were at the same speed since they all had to use the same SERDES. Which basically is 10, 14, 25/28, 32 and 50/56. Latency varies depending on the protocol. Eth is the worst obviously. QPI probably is the best. I am trying to match QPI in our cache coherent interconnect which is built over SRIO's GSM functionality. 

But as I pointed out, the issue with PCIe is its fundamentally flawed architecture model. Like our reptilian brain stem it cannot get rid of its PCI ancestry ! The attendent flaws are fixed in the genes. Remember Intel's aborted switch fabric standard over PCIe, ASI. We started on an IP on that too before quickly coming to the conclusion that it was a non-starter.

PCIe switch fabrics are like experiments with fully  socialist governments. Every now and then somebody thinks it is a good idea and tries to have a go at it. Then quickly realizing the futility of that exercise, give it up. Till somone comes along a few years later ! Unless you area  glutton for punishment, why on earth would you try doing a fabric using non-transparent bridging ?

 

Now the latency claims you are talking about I think are purely in the SW/driver domain. technically has nothing to do with the standard. But if all standards implemented optimal drivers, then the stadard's silcion latency would again be the determining factor. It is to alleviate this, that in our experimental processors we are linking the SRIO endpoint to the processor core (exactly the way Transputer did it eons ago, funny how things never change) . So a SRIO message is just a single instruction overhead. message will appear either on a special buffer or the cache in the remote cpu. That is the way to build a fast interconnect. You can bypass all the cache and MMU nonsesnse.

rick merritt
User Rank
Blogger
Re: Not your father's PCIe
rick merritt   2/26/2014 11:09:53 PM
@GSMD: So A3Cube's latency of 100 nanoseconds is the same as SRIO--so why do they bother? Game over for them?

GSMD
User Rank
Manager
Re: Not your father's PCIe
GSMD   2/27/2014 4:57:21 AM
1. I would hesitate to answer your question without knowing more about A3Cube's claims. Not sure if the latency is lower becuase of better SW stack or an optimized silicon datapath.

I guess they feel there is a good PCIe interconnect market and the best PCIe implementation will get them business. This is not bad reasoning since to use IB or SRIO in an x86 system, you anyway add the extra PCIe latency. So the latency advantages of SRIO or IB get wiped out. So even if you are on par with SRIO oor IB, there probably is good business.

2. It is also not clear as to what portion of the packet flow the 100ns latency refers to, host controller or switch. Tough to analyse given the paucity of data.


But as I pointed out, latency is not the only issue when doing interconnects, the interconnect should support peer to peer topologies. PCIe does not natively do so and hence you pay the penalty in switch latency and silicon cost.

Page 1 / 3   >   >>
Flash Poll
Radio
LATEST ARCHIVED BROADCAST
Join our online Radio Show on Friday 11th July starting at 2:00pm Eastern, when EETimes editor of all things fun and interesting, Max Maxfield, and embedded systems expert, Jack Ganssle, will debate as to just what is, and is not, and embedded system.
Like Us on Facebook

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)
EE Times on Twitter
EE Times Twitter Feed
Top Comments of the Week