10GbE controllers: Trends, requirements, and top sources

Datacenter Network Convergence

Although Ethernet has been the de facto technology for the general purpose LAN, Gigabit Ethernet has been considered as a sub-optimal switching fabric for storage networking. This is due primarily to performance issues stemming from the fact that GbE has lower bandwidth than InfiniBand and Fibre Channel, and typically exhibits significantly higher end-to-end latency and CPU utilization.

However, this situation has changed dramatically due to recent developments in low-latency 10 GbE switching and intelligent controllers that offload protocol processing from the host processor. These enhancements allow servers to fully exploit 10 GbE line rates, while reducing one-hop end-to-end latency to less than 10 microseconds and CPU utilization for line-rate transfers.

As a result, 10 GbE end-to-end performance now compares very favorably with that of more specialized datacenter interconnects, eliminating performance as a drawback to the adoption of an Ethernet unified datacenter network. Off-loading  protocol processing from the central CPU to intelligent 10GbE controllers can also improve the power efficiency of servers because ASIC 10GbE  processors are generally considerably more power- efficient in executing protocol workloads.

I/O Consolidation Using 10GbE iSCSI, NAS and FCoE

The concept behind I/O consolidation is simple: the sharing of storage and networking traffic on the same Ethernet physical cable or, in cases that network isolation is desired, the flexibility to configure and use the same hardware for both types of network loads, and the prioritizing of traffic delivery through quality of service (QoS) metrics. The benefits end-users will realize from this simple idea are significant.

Companies that leverage I/O consolidation will be able to realize significant gains in server slot efficiencies by using multi-function network/storage adapters to simplify their cabling scheme within a rack, thereby reducing the amount of heat each server generates.

The dominant approaches to storage I/O consolidation are network-attached storage (NAS) and  iSCSI (Internet SCSI). NAS uses TCP/IP-based file-access protocols, such as network file system (NFS) and common Internet file system (CIFS) to provide file-level access to file servers or file storage appliances on the IP/Ethernet network.  iSCSI is a storage area networking (SAN) protocol which replaces the FC stack with the standard networking TCP/IP stack in order to transport block storage traffic over standard lower cost Ethernet.

iSCSI SANs and NAS solutions based on the Gigabit Ethernet LAN interconnect have been widely deployed in smaller, more homogeneous data centers that are typically found at the third tier of large enterprise data infrastructures or in the data centers of small to medium sized enterprises (SMEs). With 10GbE IP storage being established as a viable alternative to Fibre Channel SANs, the stage is set for dramatic iSCSI/NAS market expansion as IP storage devices leverage the performance and cost-effectiveness inherent in 10 GbE enterprise switch/routers and 10 GbE server controllers.

The value proposition of the emerging FCoE standard is based primarily on the elimination of the expensive FC infrastructure components in datacenters, which are currently used to connect servers running high-end applications to their networked storage systems. Since FCoE requires 10GbE (with Data Center Bridging (DCB) extensions in both the controllers and the switches), its broad-based deployment is not expected till 2013 and is likely to remain a niche interconnect for the foreseeable future.

There are a number of 10 GbE controllers available that provide protocol processing offloads for iSCSI and NAS support. These controller-based protocol processing capabilities combine to greatly accelerate TCP/IP and iSCSI protocol processing by increasing data transfer throughput and I/O performance, lowering end-to-end latency and reducing CPU utilization.

10GBASE-T Adoption

Data center administrators have a number of 10GbE interfaces to choose from including CX4, SFP+ Fiber, SFP+ Direct Attach Copper (DAC), and 10GBASE-T. Today, most are choosing either 10GbE Optical or SFP+ DAC. However, limitations with each of these interfaces have kept them from being broadly deployed across the data center.

Fiber connections are not cost-effective for broad deployment, and SFP+ DAC is limited by its seven-meter reach, and requires a complete infrastructure upgrade. CX4 is an older technology that does not meet high-density requirements. For 10GBASE-T, the perception to date has been that it required too much power and was too costly for broad deployments. These concerns are being addressed with the latest manufacturing processes that are significantly reducing both the power and the cost of 10GBASE-T.

In 2010, fiber represented roughly 45% of the 10GbE physical media in data centers, but this percentage with continue to drop to approximately 10% by 2013. Direct-attach connections will grow over the next few years to roughly 45% by 2013 with large deployments in IP Data Centers and for High Performance Computing. 10GBASE-T will grow from only 4% of physical media in 2010 to 44% in 2013 and eventually becoming the predominant media choice.

Server OEMs are expected to standardize on BASE-T as the media of choice for broadly deploying 10GbE for rack and tower servers. 10GBASE-T provides the most flexibility in performance and reach. OEMs can create a single motherboard design to support GbE, 10GbE, and any distance up to 100 meters. 1GBASE-T is the incumbent in the vast majority of data centers today, and 10GBASE-T is the natural next step.

10GBASE-T PHYs moving into mass production and the availability of integrated 10GbE controllers and 10GBASE-T PHYs in the 2012 timeframe for server LOM applications is also expected to drive 10GbE migration to the desktop.

The cost and versatility of structured cabling together with the ability to auto-negotiate on 10GBASE-T PHYs so that you can support both 1GbE and 10GbE provides great investment protection.  Tens of millions of 10GBASE-T-capable ports are already installed in enterprises worldwide.  With the 10GBASE-T PHYs hitting mainstream and the price decreasing significantly to enable LOM 10GBASE-T will see a large desktop market adoption beginning in 2012-2013 and ramping starting 2014.