10GbE controllers: Trends, requirements, and top sources

10GbE Controller Product Requirements

A range of vendors offer 10GbE controller devices for use in server adapter or LAN-on-motherboard (LOM) configurations. There are several different hardware and software requirements to consider including number of ports, power per port, support bus interface, supply voltages, and package size, and software capabilities including protocol offload and operating system support.

Most new 10GbE controllers and adapters support dual ports on the network side. Some 10GbE controllers use the second port only for failover, whereas the trend is for dual-port 10GbE controllers to support active/active configurations (i.e., concurrent operation for both ports).

For host-side connectivity, 10GbE controllers support PCI Express which is the industry standard CPU-to-I/O serial interconnect in volume servers. The available 10GbE controllers support a mix of PCIe v1.1 and the second generation of PCIe, PCIe v2.0 or PCIe Gen2, which is rapidly gaining market traction.

Server vendors view the power-per-port specification as an important 10GbE product requirement to allow efficient 10GbE support in adapter and LOM designs. Thus, the specific challenge vis-à-vis power is for the 10GbE controller vendors to minimize the power consumption as newer, more functional designs are offered. Getting power-dissipation data from the 10GbE controller vendors that can be directly compared is often difficult since the vendors tend to quote their lowest power numbers - which often do not come from the same device set-up.  However, it is still instructional to compare the stated power dissipation numbers - as they could provide a good relative comparison of where the power dissipation would actually be in a real-world application.

The challenge with 10GBASE-T is that the early physical layer interface chips (PHYs) have consumed too much power for widespread adoption. The same was true when gigabit Ethernet products were released. The original gigabit chips were roughly 6.5 Watts/ port. With process improvements, chips improved from one generation to the next. The good news with 10GBASET is that these PHYs benefit greatly from the latest manufacturing processes. PHYs are Moore's Law friendly, and the newer process technologies will continue to reduce both the power and cost of the latest 10GBASE-T PHYs.

Beyond the 10G PHY aspect, a number of additional factors impact the effort required to design a 10GbE controller into a PCIe adapter or a server LOM application. Primary among these is the effective 'footprint' of the controller, including the package size and the amount of external memory required.

From a software standpoint, the important requirements are the type of protocol offloads supported by the controller as well as availability of drivers for the main server operating systems. The issues to consider from an IP protocol offload standpoint are if the TCP/IP offloads are stateless or stateful, and the performance and ease of deployment benefits of the chosen offload approach. In addition, availability of controller drivers for Linux and Windows, as well as support of the top virtualization platforms, including VMware and Xen, are important requirements.

Leading 10GbE Controller Vendors

Industry analysts estimate that Intel has achieved volume leadership position in the 10GbE controller market segment, with Mellanox and Chelsio also being part of the select group of vendors with a broad line of product offerings in this segment. Tehuti Networks is an emerging supplier of 10GbE controllers focusing on the emerging, potentially vast market for 10GbE adoption for the desktop.

Intel 82599

Intel's third-generation 10GbE controller, the Intel  82599 10 Gigabit Ethernet controller continues to build on functionality of its predecessor offerings. The Intel 82599 controller is a single-chip, dual-port 10GbE PCIe Gen2 implementation in a 25x25 mm package. It reduces design complexity by integrating serial 10GbE PHYs and provides both SFI and KR interfaces. With efficient power consumption, a small footprint, a large on-chip packet buffer for maintaining network performance and integrated PHYs, the controller is ideally suited for Server Blades, LAN on motherboard (LOM), controller, and Mezzanine card implementations.

The Intel 82599 fully supports converged networking through incorporating support for data center bridging (DCB). IN addition, it includes hardware optimizations that help reduce I/O bottlenecks and improve the overall server performance in virtualized server environments. These technologies are Virtual Machine Device Queues (VMDq) and Virtual Machine Direct Connect (VMDc) which is based on the industry-standard PCI-SIG SR-IOV (Single Root I/O Virtualization).

Customers can use the Intel 82599 to carry iSCSI, NAS or FCoE traffic over Ethernet. In order to meet SAN requirements for guaranteed packet delivery, the controller implements capabilities such as enhanced transmission selecting (ETS) and priority flow control (PFC).The controller accelerates iSCSI traffic by implementing key stateless offloads such as TCP segmentation offload (TSO) and Receive Side Coalescing (RSC). It also supports the trusted iSCSI initiators from Microsoft, Linux,* and VMware operating systems and provides an iSCSI remote boot implementation.

Regarding FCoE, the Intel 82599 10 Gigabit Ethernet controller delivers a high performing FCoE solution that offloads the main data paths for I/O read and write commands. It also reduces CPU processing on FCoE receive traffic by eliminating a data copy through Direct Data placement implementation.

Mellanox ConnectX-10GbE

Mellanox ConnectX-2 EN 10 Gigabit Ethernet Media Access Controller (MAC) delivers dual-port 10GigE connectivity with stateless offloads for performance-driven PCI Gen2 server and storage applications in Enterprise Data Centers and High-Performance Computing environments.  The device is also well suited for Blade Server LOM designs due to its relatively small package size of 21mm X 21mm, integrated XFI and backplane PHY interfaces, and no requirement for external memory.

ConnectX-2 EN is supported by a full suite of software drivers for Microsoft Windows, Linux distributions, VMware and Citrix XenServer. It supports stateless offload and is fully interoperable with standard TCP/UDP/IP stacks, as well as various management interfaces with a rich set of configuring and management tools across operating systems.

ConnectX-2 EN delivers the features for a converged network with support for Data Center Bridging (DCB). RDMA over Converged Ethernet (RoCE) technology provides efficient, low latency RDMA transport over Layer 2 Ethernet while T11 compliant FCoE support with hardware offloads simplifies storage networking. iSCSI and NAS acceleration is enabled by hardware-based stateless offload handling TCP/UDP/IP segmentation, reassembly, and checksum calculations that would otherwise burden the host CPU.

Chelsio T4

The Terminator 4 (T4) ASIC represents Chelsio's fourth-generation 10GbE controller design. For the server connection, the chip includes a PCI Express v2.0 _8 host interface. On the network side, T4 integrates four 10 GbE ports into a 27mm x 27m package.

With embedded 10Gbps Serdes interfaces, all four ports offer direct support for 10GBASE_KR. Two ports also support the four_lane 10GBASE_CX4/KX4 interfaces. In a blade_server design, T4 provides redundant 10GBASE_KR or _KX4 backplane connections without requiring external PHY devices. All four network ports support direct connection to SFP+ modules using SFI. The per-port power rating for T4 is 2W.

T4 provides broad and proven support for file_ and block_level storage. In NAS filer/head designs, T4 provides full TCP/IP Offload Engine (TOE) for Linux and FBSD_based embedded operating systems.  For block storage, T4 supports partial iSCSI data path offload, where the ASICs offload processing_intensive tasks such as PDU recovery, header and data digest, CRC generation/checking, and direct data placement (DDP).

T4 also adds support for both partial and full offload of the FCoE protocol.  Full offload provides maximum performance as well  as compatibility with SAN management software. For customers that prefer to use a software initiator, Chelsio supports the Open_FCoE stack and T4 offloads FCoE data path tasks much as it does in iSCSI.

Tehuti TN3020

Tehuti TN3020-DB is a dual-port, 10 Gigabit Ethernet (10 GbE) Network Traffic Accelerator (NTA), delivering cost effective and high-bandwidth access for LOM, NIC, server and storage applications. Targeted at enterprise data centers, embedded environments as well as low-end NAS and IP video servers and embedded environments. The TN3020-DB substantially reduces the CPU's TCP/IP packet processing tasks, while consuming small package size and low power.

It improves network performance with low power consumption (2.2W per port) and compact footprint (21mm x 21mm). These capabilities coupled with an aggressive cost per port position the TN3020-DB as a focused offering for driving 10GbE adoption in low-end storage environments.

Using Tehuti's patent-pending architecture, TN3020-DB uses stateless offload techniques to shift computation-intensive UDP/TCP/IP processing away from the CPU to the Network Traffic Accelerator (NTA) chip, providing significant network performance improvements. Tehuti's offload approach removes bottlenecks in the host's CPU, system memory, and enables a balance between the CPU, software, and the offload controller hardware.

Specific offloads include checksum operation, large packets transmission receive side scaling, which minimizes CPU utilization across multiple processor systems running Windows OS, as well as L2 services. TN3020-DB enables iSCSI and NAS acceleration through the Tehuti offload architecture and its specific stateless offload approaches for TCP/IP and UDP/IP. For FCoE, as with other stateless offload offerings, TN3020-DB enables high-performance and efficient operation of FCoE software initiators, such as the Linux Open-FCoE stack.

Tehuti and 10GbE Adoption on the Desktop:  Tehuti's focus for its next generation, single and dual ports 10Gb Ethernet Controller and Accelerator devices is on ubiquitous 10GbE adoption on the desktop with very attractive cost and a power level with less than 1W per port. Together with the latest generation of 10GBase-T PHY devices which deliver power consumption of approximately 2.5W, a power rating of 3.5W suitable for an integrated controller/PHY combination for LOM applications can be achieved.

The three key enablers for pervasive deployment of 10GbE on the desktop that Tehuti's next-generation controllers promise to address are CPU performance, per-port cost and power consumption  for 10GbE LOM. High-performance CPUs are already available on most desktop PCs which dual, triple and quad core solutions at an affordable price. These latest multi-core desktops can efficiently utilize 10GbE network while running other user applications. Ethernet storage applications (NAS and iSCSI) are widely used on desktops and are projected to be one of the main 10GbE desktop applications.

The cost of a 10GbE LOM solution for enabling 10GbE use on desktops is another enabler that Tehuti is focused on. As discussed earlier, starting in 2012-2013, PC manufactures will start offering desktops with cost-effective 10GBASE-T LOM solutions fully backward compatible with the current 1000BASET and which out-of-the-box can run on existing structured cabling.

LOM power is final barrier for desktop adoption of 10GbE. With today's solutions running at ~6-8 Watt per port (including controller and PHY), making a robust 10GbE LOM solution require active cooling or high-cost heat sinks. While this may be acceptable for server adapters, it is not a viable solution for server or desktop LOM applications. A total per-port power budget of 3.5W for both controller and PHY must be met. This requires the 10GbE controller to support a 1W/port power rating which is the goal for the next-generation Tehuti controller.

About the Author

Saqib Jang is Principal at Margalla Communications, a Woodside,CA-based firm providing strategic and technical marketing   services to the storage & networking markets.  He has several years of experience as a successful storage networking marketing consultant and industry analyst, and over fifteen years of experience in the product planning, development, marketing, and business development of industry-leading storage networking products.  Mr. Jang holds a B.S. degree in Electrical Engineering from Massachusetts Institute of Technology (MIT) and an M.B.A. in Marketing from the Wharton School, University of Pennsylvania. He can be reached at saqibj@margallacomm.com