Engineers want to remain agnostic. Whatever high-speed interconnect works best for a given design is what they will use, they say.
But that doesn't mean they don't have some pretty strong opinions, especially since many of them spend lots of time working on or tracking the growing pack of often-competing interconnect efforts. In our focus groups and individual interviews, engineers said:
- PCI Express is happening in a big way as a chip-to-chip interconnect.
- 10-Gigabit Ethernet is the leading contender for next-generation backplanes. Infiniband, though seemingly waning, remains a dark horse.
- Parallel PCI is waning and HyperTransport is wavering as chip-to-chip links.
- Xaui and Systems Packet Interface chip-to-chip interconnects are holding their own.
- Many engineers are not familiar with Advanced Switching (AS) and RapidIO, though some are beginning to move toward AS.
Among new efforts, PCI Express is coming on strong. Express is nearly as well-known and popular an interconnect as Ethernet, according to a straw poll of focus group participants drawn from a cross section of communications and computer companies. As many participants made decisions to use Express as Ethernet in the last two years, and Express and Ethernet are tied when it comes to the interconnect most participants expect to use in the next two years.
"To some extent, I/O standards are like [CPUs] because it's very difficult to switch. No one wants to be the first chicken to go and no one wants to be the last chicken," said one focus group participant.
Engineers from Apple, Cisco, Hewlett-Packard, Juniper Networks and startups like Tidal Networks all said they see a significant shift to PCI Express in the works, and they expect to use the interconnect in future systems.
Among the stalwarts, Ethernet still reigns. The Xaui derivative of Ethernet has lots of legs in chip-to-chip and backplane interconnects, said engineers from Cisco Systems Inc. and Nortel Networks Ltd., where Xaui is a legacy technology.
"Ethernet is the dominant protocol because it is so cost-effective," said Michael Krause, an interconnect specialist in HP's server group. "Things like Advanced Switching and serial RapidIO just aren't on our radar screens. We evaluate them but they are not very interesting for us."
Krause is working to drive advances in Ethernet as part of the iWARP effort that is bringing such performance improvements as lower latency to the widely used technology.
Ethernet's new tricks
As one example of how Ethernet is improving, Krause said one company has taped out an unannounced multiport 10-Gigabit Ethernet switch that sports latency of 200 nanoseconds. That's far below the 1-microsecond latency of many Ethernet switch chips and nearly on par with the 100-ns latencies of Infiniband, seen by many as the cream of backplane interconnects.
Engineers from Network Appliance Inc. and Sun Microsystems Inc. said they hope 10-Gigabit Ethernet could become a dominant backplane interconnect in a couple of years-if current standards efforts pick up their pace.
Network Appliance implemented Infiniband for storage systems several years ago when it needed a standard high-performance backplane. At the time, no version of Ethernet was up to its throughput and latency needs.
"We had no choice and shipped Infiniband in anger," said a NetApp engineer. "For a long time we were Mellanox's biggest customer. We wanted Ethernet to step up to the plate, but it was dillydallying."
Infiniband "doesn't have as broad an ecosystem as we'd like to see," he said, adding that NetApp is also using HyperTransport as a chip-to-chip link and tracking Advanced Switching as a future backplane contender.
Given its strong road map to 40-Gbit/second links, however, Infiniband still has potential as a dark horse and could leapfrog Ethernet.
"Infiniband has lost a lot of momentum since the downturn, but it's still happening. These days we see it as more of a systems-to-system interconnect than a holy grail of fabrics," said one computer interconnect engineer who asked not to be named. "AS is still new and we are not sure about it yet."
"Infiniband was like the dot-com bubble. It burst," said one Sun engineer, who said 10-Gigabit Ethernet is now his favorite candidate for a next-generation backplane.
"The goal is the $50-per-port 10-Gigabit backplane with full crossbar switching, and whoever gets there first wins," said the NetApp engineer.
As serial PCI Express is on the rise, its grandfather, parallel PCI, is on the wane. Though it will no doubt be used for years-if not decades-it has effectively hit the end of its technology road map (see related article, page 38).
The other major parallel chip-to-chip link, HyperTransport, appears to be wavering-or worse. In a recent panel discussion, Sean Smith, a former verification engineer at Cisco Systems who is now with Denali Software Inc., suggested that big OEMs like Cisco are turning away from HyperTransport.
"Cisco was a very large proponent of HyperTransport-[when we] were running out of bandwidth and needed an alternative. It served its purpose for a number of people for a number of years, but, personally, I don't see it competing very well against these newer technologies like PCI Express and RapidIO, where there is expected to be wider availability of low-cost silicon support," Smith said.
In our focus group, participants said they were relatively familiar with HyperTransport. A few had made decisions to use the interconnect in the recent past, but the group was about equally divided between those likely to use it and those not likely to use it in future designs.
Draft spec gains
In contrast, Advanced Switching-a derivative of PCI Express geared as both a chip-to-chip and backplane interconnect-is showing signs of gaining some traction, though it's still far back in the pack.
Advanced Switching was slightly better known than HyperTransport among participants and almost as well known as Infiniband. That's surprising, given AS is still a draft specification with no supporting silicon while HyperTransport and Infiniband have been used in shipping silicon for several years.
In addition, more engineers made decisions to use AS in the past two years and expect to use it in the next two years than have or expect to adopt RapidIO, an interconnect closely linked to the PowerPC and some DSPs. RapidIO was the least familiar interconnect to our focus group participants.
Both AS and RapidIO hope to replace interfaces like System Packet Interface and become a converged data- and control-plane connection in future comms systems. Both have a considerable marketing job, however, if they are going to attain that goal.
Several participants in our focus group said that they were not familiar with AS. Even more of them expressed doubts about RapidIO. Engineers from Cisco and Juniper said they had looked at RapidIO a year or more ago but haven't stayed current on the technology. The NetApp engineer said that they looked at AS as part of a PowerPC evaluation, but ultimately chose not to use the PowerPC.
The increasingly important serializer/deserializer is one of the few interconnect areas currently not getting enough standards attention.
"I would love to have an IEEE standard for a 5-Gbit serdes. Right now this is left up to the ASIC and chip vendors," said the NetApp engineer.
An IEEE standards group and an ad hoc vendor effort are currently trying to flesh out 10-Gbit serdes standards. And a PCI Express working group is deliberating whether it will adopt 5- or 6.25-Gbit signaling for its next-generation standard. A decision is expected by February and could drive a lot of serdes designs.
Vendors drive it
OEM engineers have two problems with the many interconnect standards efforts. They require lots of work-writing, reviewing and promoting specs-and the efforts are too driven by the agenda of chip makers.
"The people who develop standards don't put the needs of the people who consume those standards first. They are vendor-driven," said Chuck McManis, a technical director at Network Appliance, speaking in a one-on-one interview.
Mary Mandich, a technical manager at Lucent Technologies, agreed. "Unlike many situations, standards for serial backplane are running ahead of market acceptance," she said.
"The reality is Intel is going to optimize its processors to work with PCI Express and AS," said McManis. "To the extent they are successful and drive volume sales they could eat into the comms business, which has historically been the domain of Motorola with its PowerQuicc and Power processors that use RapidIO. The one barrier to RapidIO being successful is that the world's largest processor maker-Intel-is not in their camp," he said.
Despite all their opinions, OEMs try to remain agnostic about interconnect standards, adopting what makes sense for each particular design.
"We have no bias about a single standard. Cisco is made up of many groups and subcompanies and there is no religion here on [interconnect] standards," said Nader Vasseghi, a director of advanced engineering in Cisco's router group. "Delivering the best products we can is what's important, and that means sometimes we use PCI Express, HyperTransport and so on," he said.
Exactly which interconnect is chosen for which products depends on many issues, including time-to-market, performance, legacy interconnects, size of packet payloads, costs, data rates and more, Vasseghi said. Sometimes, the software engineer makes the call by specifying a processor-whatever interconnects it uses-based on whether the processor runs the product's legacy code, another Cisco engineer said.
Some chip makers warn, however, that hardware designers cannot become complacent about which higher-level interconnects they adopt. Each has unique implications that extend down to the I/O transistors of a physical-layer (PHY) chip.
For example, IBM Corp. did an extensive comparison of Infiniband and PCI Express and found some 10 ways the PHY chips for the two protocols were different. That's despite the fact that both chips essentially tried to use the same 2.5-Gbit technology, said Al Yanes, a chip set designer at IBM.
Similarly, after analyzing the protocols closely, Rambus Inc. found that it could design a single PHY that would cover three of the interconnects. However, that design had to be a new block that was a hybrid of existing PHYs in the company's library, said Kevin Donnelly, who manages the logic interface group at Rambus.
See related chart