Click here for Part 1.
Why create Convergence Enhanced Ethernet?
The industry has pursued several fabric convergence options, including InfiniBand (IB) and iSCSI. Each of these options has done well in a segment of the IT market. However, none has been able to fully satisfy enterprise data center storage requirements.
Enterprise data center storage customers are conservative when it comes to deploying new storage standards. Most want the new standard's ecosystem to be fully available and, more importantly, vetted out, before they deploy it. In other words, for any given new standard initiative, customers want all the basic capabilities (e.g. reliability, availability, serviceability, performance) to be robust enough to meet their application environment's demands. Additionally, these customers have a large install base of Fibre Channel components (servers, storage and switches) and want to protect that investment.
We will cover how IB, iSCSI and Fibre Channel over Internet Protocol (FCIP) compares to Fibre Channel over Convergence Enhanced Ethernet (FCoCEE) later in this article. But first we will describe why Convergence Enhanced Ethernet (CEE) was needed, versus just using Fibre Channel (FC) over (today's) Ethernet.
To satisfy performance requirements, storage fabrics require a lossless traffic service. Today's Ethernet can provide lossless behavior through the use of the IEEE 802.3x Pause standard. This standard allows a congested Ethernet port to assert back-pressure through an XON/XOFF protocol. Using IEEE 802.3x, the receiver will send an XOFF when it can't take any more data. The sender will then pause until the receiver sends an XON to resume communications. IEEE 802.3x works fine for Ethernet networks that are dedicated to storage traffic, because congestion only affects the storage traffic.
If Fibre Channel were to run directly over an Ethernet link that is also carrying other traffic classes (e.g. LAN or HPC), the IEEE 802.3x pause mechanism causes head of line blocking for all traffic classes flowing over the congested fabric segment. For example, if a storage flow experiences periods of traffic bursts that cause congestion, the congestion spreads to other traffic classes, such as sockets communications between servers or between servers and clients. Obviously, performance collapse is not acceptable for a convergence fabric. Over-provisioning can always be used, but is more expensive and must be continually upgraded to stay far above the possible IO demands.
The industry is working on enhancements to Ethernet that provide per priority flow and congestion control. The standard version of these enhancements is known as Convergence Enhanced Ethernet (CEE). For customers seeking FC and Ethernet convergence, FCoCEE is the model we advocate.
What is Convergence Enhanced Ethernet?
IBM is working with Broadcom, Brocade, Cisco, Emulex, Intel and several other companies to create a set of IEEE 802 standard functions that improve Ethernet's ability to converge fabrics in the data center. Convergence Enhanced Ethernet is the term used to refer to the IEEE 802 standard version of these functions.
CEE consists of the following enhancements being pursued in IEEE 802 working groups:
- Multiple priority levels with per priority flow control - this effort is focused on creating a standard mechanism that can control the flow of a single traffic class, without affecting the flow of other traffic classes on the same link. With per priority flow control when a traffic class gets congested, other traffic classes running over the same link are not affected. Additionally, it can be used to provide lossless transmission for specific traffic classes.
- Priority based packet scheduling mechanism - this effort is focused on creating a standard mechanism that can be used to set scheduling priorities for a set of traffic classes. The scheduling mechanism provides end-to-end QoS. The scheduling policy isn't degraded by competing traffic in the network (e.g. allowing small messages to be inserted between the packets associated with a large data transfer).
- CEE discovery and capability exchange protocol - this effort is focused on creating a standard mechanism for defining the domain formed by CEE compliant components in order to ensure interoperability.
As shown in Figure 6, the functions described above allow CEE to provide traffic differentiation, such that multiple traffic classes can flow over the same link, without impacting each other. CEE can be combined with iSCSI to provide traffic differentiation at the Ethernet link layer for iSCSI data flows. For example, network adapters can associate a specific traffic class with iSCSI traffic and a different traffic class for application sockets communications. CEE can then be used to provide service differentiation between these two traffic classes.
6. CEE Overview