1-Gbit and even 10-Gbit Ethernet is in the offing, with at least two board-level products almost ready for deployment
Material adapted from reports prepared by EE Times editors Rick Merritt and Loring Wirbel
Startup company Ammasso Inc. (Boston, Mass.) is working on an enhanced Ethernet for the data center product. Ammasso's 1100 card, a 1-Gbit Ethernet adapter for server clusters, supports a TCP offload engine (TOE) and remote direct-memory access.
TOE and remote DMA are seen as key technologies to reduce latency over Ethernet, opening the door to its use as a ubiquitous networking technology in the data center, replacing proprietary server-clustering technologies and relatively expensive Fibre Channel networks for storage-area networking.
Top OEMs Are Driving Force
Several top data center OEMs are driving the remote-DMA specification through the RDMA Consortium, and chip and card startups supporting the Consortium's work are emerging in a wave following the spec's completion.
For example, Infiniband startup Banderacom renamed itself NetEffect Inc. (Austin, Texas) and reorganized to build Ethernet silicon with RDMA and TOE.
Established players, including Alacritech Inc. (San Jose, Calif.) and Broadcom Corp. (Irvine, Calif.), have announced Ethernet chips with RDMA and TOE.
TCP offload is also being supported natively under Microsoft's upcoming versions of Windows.
Initially, Ammasso is targeting only clustering because it lacks support for the iSCSI standard to handle block storage access over Ethernet. The company has pinned its hopes on the so-called iSCSI Extensions for RDMA.
Now in development at the RDMA Consortium, the extensions aim to blend iSCSI support into the RDMA spec. It's not clear when the work will be complete.
A number of companies have stumbled in efforts to pack 1-Gbit Ethernet, TOE, RDMA, and iSCSI into silicon. In part, that's because their chips proved too expensive to compete with Fibre Channel, which is already shipping at 2-Gbits/s and is rapidly moving to 4-Gbits/s.
Ammasso aims its 1100 card at server clustering, providing native support for the Message Passing Interface used in technical applications and the Direct Access Provider Library used in commercial clusters. It's still optimizing performance on the product, targeting an application-level latency of 10-µs. The card will soon be available for about $825 in single-units, with OEM volume discounts.
Waiting For Standards Ratification
The 1100 is initially based on a Xilinx Virtex Pro FPGA (field programmable gate array). However, the company has started work on an ASIC that will support both 1-Gbit and 10-Gbit Ethernet, but it's waiting for RDMA standards to be ratified at the Internet Engineering Task Force before it commits to silicon.
Meanwhile, newcomer Ethernet specialist Chelsio Communications Inc. (Sunnyvale, Calif.) is getting set to introduce a card that can handle 10-Gbit Ethernet traffic at wire speed with 10-µs latency. Its PCI-X card, due for launch at the Grid Today 2004 conference in Philadelphia, is based on a single-engine ASIC dubbed the Terminator. It has a TCP offload engine optimized for grid computing and data center applications.
Chelsio says its T110 card will handle full TCP offload and will offer full-duplex performance at 10-Gbit/s speeds. It's designed for fiber short- and long-reach applications with Xpak transceiver modules, but will also handle coaxial interconnect utilizing Marvell Semiconductor's CX-4 interface (using 4X twinax cabling).
Chelsio's founders came from SGI's Protocol Engines Inc. Subsidiary, and were responsible for designing the SGI Origin supercomputer. They understand the bottlenecks of using Ethernet framing as a channel-based alternative to interconnects such as Infiniband. They also learned about the value of retaining standard Layer 4 protocols such as TCP (Transmission Control Protocol).
"You can develop a wonderful alternative technology, but if it doesn't fit the framework of standards, it doesn't gain a critical mass," says the company's CEO Kianoosh Naghshineh.
The team decided to offer a deliverable as a host bus adapter rather than a chip set, since most server and storage OEMs are familiar with HBA. Nevertheless, Chelsio says it has already entertained purchase requests for its Terminator ASIC. Terminator can handle 1 million concurrent TCP sessions, although the card is configured for no more than 64,000, indicating the future scalability of the architecture. Terminator has more than 5 million equivalent gates, implemented in a 0.13-micron process.
Several large Ethernet players, including Intel and Broadcom, have sampled 1-Gbit TCP offload engines, but Chelsio claims an additional benefit besides a 10x speed advantage. Chief technology officer Asgeir Eiriksson says that no aggregated RISC engines are used in the architecture.
A VLIW Architecture
Instead, Terminator is based on a VLIW (very long instruction word) architecture in which no data is cached. Control-plane CPUs hence show very low utilization rates when used with the T110 host bus adapter (less than 50% with a 2.2-GHz uni-processing Opteron, and less than 15% with a 1.5-GHz uni-processor Itanium).
The board can achieve 7.8-Gbits/s when standard Ethernet frames are used, and it also supports the Jumbo Frames format.
Eiriksson says the Terminator architecture permits better scaling at both low- and high-ends. Multiple RISC engines can only obtain full wire-speed performance when 10 channels are used. The Chelsio board can achieve full speed with as few as two channels, but it doesn't run into cache thrashing when multiple channels are used.
At the upcoming Grid Today conference, Chelsio will demonstrate 7.8-Gbit/s performance with 9.5-µs application latency over Hewlett-Packard's Integrity and Opteron servers connected via a Fujitsu switch with fiber and copper interconnect.