In August of 1999 all of the major server vendors combined to develop a new I/O "fabric" for servers and storage systems within the enterprise data center called the InfiniBand Architecture. This fabric's goal was to provide IT managers with a better way to deploy, scale, and manage computing power within the data center.
Today, the InfiniBand architecture is meeting this goal and is the only 10Gb/sec transport that enables industry standard servers to be clustered together for reliable, available, scaleable and high performance enterprise computing. InfiniBand and Linux are the perfect marriage for distributed, industry standard computing. InfiniBand connected system-to-system fabrics deliver lower capital and operating costs by enabling the following:
Industry standard servers and storage systems can be clustered together without the cost or proprietary lock-in of vendor specific solutions.
InfiniBand accomplishes this at 10Gb/sec with ultra low clustering RDMA latencies that enables break through performance for both enterprise data base applications and High Performance Computing (HPC).
InfiniBand products, featuring 2nd generation silicon, are available today from many suppliers.
Record Performance on InfiniBand
Enterprise Data Bases: In April 2003, Oracle released some impressive performance results with InfiniBand and committed to include native InfiniBand support in their next major release. The Oracle white paper, Achieving Mainframe-Class Performance on Intel Severs Using InfiniBand Building Blocks, states that InfiniBand provides a two to four times performance improvement for Oracle 9i Real Application Clusters (some results shown below in Figure 1). InfiniBand's combination of high bandwidth, low latencies and low CPU utilization will provide IT managers with a significantly improved return on investments and reductions in TCO.
Figure 1: InfiniBand and Gigabit Ethernet compared. Source: Oracle
HPCC: The High Performance Compute Clustering (HPCC) market is a vital IT requirement for many companies. These clusters are used for such applications as: complex financial analysis, automotive design and crash tests, seismic simulations in gas and oil research, bioscience or molecular modeling, industrial simulations, CAD, graphics rendering, and others compute intensive tasks. HPCC might not seem like a standard Enterprise Data Center (EDC) application but many corporate data centers feature these applications as they represent the core R&D function within many companies.
In addition, HPCC is a great bellwether for Enterprise Data Center performance. Initial benchmarks available have demonstrated 8X the bandwidth vs Ethernet delivered with an order of magnitude lower latencies. MPI over InfiniBand has shown better latencies (sub 7 µsec) and greater bandwidth than that of the proprietary HPC clustering technologies (Figures 2 and 3).
Figure 2: MPI bandwidth vs. message size. Source: Ohio State University, Department of Computer and Information Science
Figure 3: MPI latency vs. message size. Source: Ohio State University, Department of Computer and Information Science
InfiniBand, Always a Generation Ahead
Microprocessors and industry standard servers continue to advance at a rapid pace and InfiniBand is the only I/O technology that can keep up with these advancements. Not only is InfiniBand available at 10 Gb/sec today, but the specification for 30 Gb/sec (12X) InfiniBand links has already been approved and silicon devices will be available soon. All InfiniBand products feature Kernel bypass support, RDMA support, hardware transport, and management semantics that make InfiniBand the technology and performance choice for enterprise data centers. These advanced features and unmatched performance ensure that InfiniBand based products will match the bandwidth of the processors of the future and stay a generation ahead of the competition.
To provide the utmost flexibility Mellanox has designed a modular HCA that enables rapid local bus adaptation to match future server chip sets, whether it be PCI-X 2.0, PCI Express, or any other local interface. Today, Mellanox InfiniHost HCAs feature PCI-X with PCI-Express and PCI-X 2.0 under development as the server interconnect options over the next year (for more information, see Mellanox's IDF Spring 2003 presentations). This flexibility of design will enable InfiniBand connections to all the major server chip sets scheduled for 2003 and 2004.
When it comes to competitive technologies, there are three attributes that favor InfiniBand:
Time-to-Market: InfiniBand technology is a generation ahead. Second generation 10Gb/s RDMA InfiniBand products are in the market today. By contrast, the first Ethernet RDMA specification won't be completed till fall 2003, with first generation products due in 2004. By that time 12X or 30 Gb/sec third generation InfiniBand products, again with RDMA, will already be deployed in the market.
Cost: Multiple vendors have introduced, interoperable10Gb/sec InfiniBand switches with port costs from $600 to ~$1000 per port. These 10Gb/sec InfiniBand products can currently be purchased. InfiniBand dual port 10Gb/sec host channel adapters are already in the market at $999. By contrast Ethernet RDMA products are due till 2004, and the current 10 Gb/sec Ethernet NICs are $7,995 each with switch ports starting at $12K.
Complexity: InfiniBand was designed specifically to support low latency RDMA semantics for clustering. Other technologies, such as Ethernet, must overcome legacy issues to support this critical technology. This is a key reason why InfiniBand will always cost less than 10 Gb/sec Ethernet with RDMA and a TCP offload engine.
Markets for InfiniBand
The Enterprise Data Center is the primary target market for InfiniBand products but the flexibility and very nature of the architecture allows InfiniBand to address many different market segments. InfiniBand is being used in applications that require a reliable and resilient fabric or just a high-speed low latency interconnect. Markets where InfiniBand is viable today, include:
Enterprise Data Center (EDC): Major enterprise data bases, like Oracle and DB2, support many of the current enterprise IT advancements and will support InfiniBand. Also, this market includes InfiniBand as an interconnect technology for server blades.
Native attached InfiniBand storage: NetApp and Fujitsu are leading a high performance easy to use file based InfiniBand storage effort.
High Performance Computing Clustering (HPCC): Utilizing a number of industry standard servers for HPC clustering for compute intensive corporate R&D or academic research efforts.
Embedded Applications such as the long standing VME specification, PICMG standard, and other applications like video streaming, military, storage fail-over, Advanced TCA for Telco and others.
There will be four major phases for InfiniBand capability deployments into the market place. They are outlined in the order of deployment:
HPC clusters will be deployed in mid 2003. These clusters are straightforward to deploy and the software needed is ready. There are multiple versions available of an InfiniBand aware protocol called Message Passing Interface (MPI) that allows existing HPPC applications to take full benefit of InfiniBand performance without modification.
Deployment of clusters of industry standard Linux servers that feature huge amounts of raw processor and memory performance running enterprise databases that are deployed with lower capital expenditures and enable lower operations costs. These clusters can be thought of as: "how to get ALL your database into memory without a lot of expense." These clusters will soon be complemented by native InfiniBand DAFS storage and accompanied by existing I/O bridges for communication within the rest of the data center. Offering clustered database solutions at a fraction of the cost of mid-range and high-end server solutions is the key for InfiniBand achieving a solid beachhead in the data center. This begins to take hold in late 2003 and early 2004.
Major server OEMs replace their existing internal server clustering technologies with InfiniBand technology. In the past, the major OEMs have made huge engineering and monetary investments to develop, advance and deploy scalability and reliability capabilities within their own proprietary mid-range and high-end server systems. Expect these transitions in 2004.
As the Data Base and mid tier applications dependencies grow to utilize InfiniBand and expand the volume of the market, third generation products will drive down the cost of InfiniBand and expand the infrastructure. This maturity will prove the event driven self-healing capabilities of InfiniBand and expand the use of the technology in the Enterprise Data Center from the data base tier, to the applications tier and finally to interconnect with the edge of the data center.
Concurrent to the four major phases, embedded designs have been deployed into production today by companies including Network Appliance, Sky Computers, SBS Communications, Themis Computer, Ramix, and VMIX. Many others will be deployed in 2003 and 2004.
InfiniBand technology is designed for the enterprise data center and its aggressive cost requirements. Today 10Gb/sec InfiniBand technology is less expensive than current Fibre Channel technology products that deliver 20% of the performance. Volume and new InfiniBand products in 2004 will further drive down prices. InfiniBand HCA cards are less than $1000 today and the switch technology is as low as $600 to $1000 a port and driving towards cutting that significantly by the end of 2004.
Host Card Cost
10 GB Ethernet
$7995 (without RDMA)*
Not Stated (1 GbE > 60 µsec.)
From $12K (per port)
2 GB Fibre Channel
$999 to $1400
$600+ (std.), $2200 (director class)
10 GB InfiniBand
< 7="" µsec.="" (over="">
$600-$1000 (per port)
Table 1: This table summarizes price-performance expectations of 10GE, Fibre Channel, and InfiniBand. $7995 Intel 10GE card, 1GbE NIC latency from NCSA, 10GbE switch costs: Riverstone, Cisco and Force 10. 10GE latency Force 10. Fibre Channel prices and latency various Web sources. InfiniBand HBA Voltaire, latency Ohio State Univ., switch latency Mellanox.
The current 10GE switch products are $12K (or more) for each port! But, this may be somewhat misleading as 10Gb/sec Ethernet is not a data center technology but instead designed for long haul WAN connections. But many still contend that this technology will migrate from the WAN/MAN to much higher volume server and desktop computers. But due to the high cost, complexity and the lack of backwards compatibility, 10GE technology is not expected to be in personal computers any time soon, if ever. Thus the cost curve is not likely to follow the high volume path that 100Mb or 1GB Ethernet has enjoyed in the past. Adding RDMA and TCP offload engines will makes 10GE even more complex and costly.
Again, InfiniBand includes hardware transport, kernel bypass and RDMA that all results in low processor utilizations for InfiniBand. When running data base applications that charge licensing fees by processor, the IT manager wants every possible CPU cycle used on the application, and not on the transport.
Winning Technology for Today
InfiniBand is being deployed and shipped today in embedded applications including the Network Appliance FAS 900c series storage system, military implementations and video streaming applications. Early deployments of InfiniBand HPC clusters are delivering significant performance advantages over other clustering technologies. Finally performance results of enterprise data center applications running over InfiniBand have been shown with Oracle indicating a significant performance boost with InfiniBand. And there is major support from IBM, Sun, Dell, Intel and many others companies for future products featuring InfiniBand technology. Not to mention that IBM has already qualified and tested InfiniBand products for the IBM x-Series servers in their "server proven" program. This, combined with the great line up of products from the InfiniBand companies including InfiniCon, InfiniSwitch, Topspin Communications, Voltaire and others mentioned, provide all the needed components to deploy InfiniBand fabrics today.
In the near future the combination of InfiniBand, Linux, industry standard servers and clustered enterprise data bases will make a huge impact on how servers are purchased and deployed, how they perform, as well as operate within enterprise data centers.
About the Author
Byran Longmire is the Director of Product Marketing at Mellanox Technologies. He can be contacted at 408-916-0020.
David Patterson, known for his pioneering research that led to RAID, clusters and more, is part of a team at UC Berkeley that recently made its RISC-V processor architecture an open source hardware offering. We talk with Patterson and one of his colleagues behind the effort about the opportunities they see, what new kinds of designs they hope to enable and what it means for today’s commercial processor giants such as Intel, ARM and Imagination Technologies.