Is high-performance Ethernet an oxymoron?

Storage systems and high-performance computing systems have traditionally been interconnected with proprietary or highly specialized protocols like Fibre Channel or Infiniband.

Lately, however, there has been an increasing drum beat over the concept of the "unified wire" or "converged port". That is, a single technology that can be used from work groups to the data center, for general-purpose networking, storage and high-performance computing.

The idea is that these performance-sensitive applications can be run over Ethernet-TCP/IP enabling organizations to benefit from "Ethernet economics" deriving both capital expense (CAPEX) and operational expense (OPEX) savings. Can Ethernet move into the data center, the domain of high performance and specialized protocols? A number of developments are converging to make this a very real possibility.

At one end of the spectrum, the network landscape consists of high-performance interconnects like Infiniband and Fibre Channel. These interfaces are characterized by high throughput and low latency and are essentially optimized for a single purpose. In comparison to Ethernet, however, their deployment is limited. They therefore do not achieve the economic benefits resulting from ubiquity.

At the other end of the spectrum, we find the omnipresent Ethernet. With Gigabit Ethernet and now 10-Gbits/ data rates, Ethernet has the potential to satisfy the throughput demands of high-performance applications. With a single 10-Gbit/s port or multiple Gigabit ports, the Ethernet network can saturate a host's PCI-X bus.

Furthermore, physical-layer technologies like CX-4 and multi-mode fiber with electronic dispersion compensation are forecast to drive 10-Gbit/s transceiver prices down to the low hundreds of dollars from today's low thousands. At those prices, some of us may even have 10-Gbit/s Ethernet in our homes! This low price makes high-performance mass market exactly the dynamic described by Ethernet economics.

With the hurdles of throughput and even price removed, can Ethernet enter the domain once reserved for specialized interfaces? The critical hurdle that remains is application throughput. Generally, systems continue to process Ethernet traffic in the host CPU. This is akin to the days of computing prior to math co-processors or graphics co-processors. Still, the approach has sufficed until now. Previously, network data rates were a fraction of CPU speeds and CPU speeds continued to maintain a gap as described by Moore's Law. However, with multi-port Gigabit Ethernet and particularly 10-Gbit/s, Ethernet has dramatically reversed that paradigm.

A number of thoughtful studies have shown how the effect of fast networking speeds (fast relative to host processing speeds), combined with the communications intensity of today's applications and the movement of data by the host CPU to and from the kernel and application memory, causes a dramatic decline in application throughput. For a given performance level, this would necessitate additional host CPUs and possibly require additional software licenses and further management overhead. This is Ethernet economics in reverse: higher CAPEX and higher OPEX!

In order to truly leverage the benefits of multi-gigabit Ethernet in high-performance systems, the application throughput problem must be addressed. Systems cannot continue to process Ethernet in software. The host CPU must be relieved to focus on applications. Initial approaches to address this problem were found in TCP/IP off-load engines (TOEs). While TOEs can provide performance gains, protocol off-load is not sufficient. TOEs can make TCP operate faster, but not necessarily better, for high-performance applications.

Addressing the whole problem requires eliminating unnecessary data movement within host CPU memory and reducing the message latency between systems. This dictates a multi-part solution consisting of protocols, software and hardware.

Part A of the solution consists of emerging protocols that are designed to take Ethernet (more accurately TCP/IP) into performance-oriented applications. Examples include iWARP for Remote Direct Memory Access (RDMA) over TCP/IP and the well-known iSCSI for block storage over TCP/IP. Defined by the IETF (Internet Engineering Task Force) the iWARP protocol is a key development for Ethernet in high-performance applications. RDMA technology enables data to be moved from one system to the memory of a remote system without involving either operating system. This eliminates extraneous memory copies and context switches, thus improving application throughput by freeing up the memory bus and CPU cycles.

RDMA is a new concept only to Ethernet. It already exists today as a basic aspect of Infiniband and has been around for over a decade in various forms. Clearly, iWARP/RDMA is a fundamental building block. Indeed, iSER is iSCSI Extensions for RDMA, enabling iSCSI to take advantage of RDMA capabilities.

Part B of the solution consists of software that can make use of the new capabilities offered by these protocols. Lack of application support has killed many great technologies. Fortunately for Ethernet, Infiniband and others have paved the road, particularly in the case of RDMA.

Industry standard APIs (Application Programming Interfaces) are available today to leverage RDMA capability. These include MPI (Message Passing Interface) for low latency messaging used in high-performance computing, and DAPL (Direct Access Provider Library). KDAPL and UDAPL provide APIs for the kernel and user (application programs) respectively. Linux has KDAPL support and we should expect to see other operating systems that do so as well — if they don't already. Although RDMA is broadly adopted in high-performance computing, commercial applications are fewer, though there are some — the most notable being Oracle 10G. However, with direct operating system support today applications can at least benefit indirectly as functions like NFS take advantage of RDMA.

Finally, Part C of the solution is a co-processor that assists the host CPU in making use of these new protocols and APIs. The role of the co-processor is to provide full transport protocol offload and processing of upper protocols such as iWARP, iSCSI and/or iSER. The notion of a co-processor to address this problem is a new concept, but like RDMA, actually only novel to Ethernet. Infiniband and Fibre Channel, for example, have dedicated processors to relieve the host CPU from protocol processing and perform direct data placement.

To that end, we are now seeing the emergence of "Advanced Ethernet Processors" or "Next-generation Ethernet Controllers" to fulfill this co-processor role for Ethernet. These co-processors interface with the kernel and user applications through the standard APIs.

Today, Advanced Ethernet Processors are being embodied into what have been described as Intelligent NICs (iNICs) or "converged" NICs. As prices inevitably drop for these devices, and as the capability gains a foothold and broader appeal, we should see Ethernet Processors follow the same migration path as NICs, finding themselves directly on motherboards.

The ecosystem for enabling Ethernet to enter the domain of high-performance applications is coming together, creating a compelling story. Setting the stage is Ethernet's gigabit data rates, most notably 10 Gbit/s Ethernet — the entry ticket. Solidly established and prices are already on a steep decline. Next are the protocols, APIs and hardware that actually mirror what is in place today for high-performance applications. No re-invention of the wheel, but instead, a wise dovetailing.

There is broad support by top tier systems vendors, top tier sub-system vendors, operating systems and device companies both established and start-up. With these elements in place, at a minimum Ethernet will have an important impact on the performance enterprise applications going forward, effectively blurring the line between high performance and general purpose.

Charles Chi is president and CEO of Siliquent (Mountain View, Calif.)