The converged fabric (management) level uses a single fabric type, such as InfiniBand or Ethernet, but retains multiple physical fabrics. This approach still requires up to 3 (4 if a dedicated server management fabric is used) times the number of components, but consolidates to a single fabric management solution.
The converged link level uses a single fabric link, such as IB or Ethernet, to carry all the various traffic types used in the data center. Obviously the links may be duplicated for redundancy, but the key attribute is that each link carries multiple traffic types, such as storage, LAN, IPC, or management. This approach is the least expensive, because it consolidates all fabric components, including adapters, switches, cables and management. It is important to note that under this approach multiple fabric management levels may still be used. For example, an FCoCEE approach may provide two fabric management levels: a physical CEE fabric manager that manages the allocation of CEE resources to traffic types and a virtual FC fabric manager that manages the FCoCEE and FC components attached to the fabric.
Current Convergence Options
Several fabric convergence options exist today. The major ones are: InfiniBand and Ethernet, combined with NAS and iSCSI.
Figure 4 shows IB-based fabric convergence. This option uses IB-based adapters to connect servers to an IB fabric. The storage can be accessed through block or file mode IO and attached directly to IB. Examples include Engenio's 6498 and Data Direct Network's S2A9550. The storage can also be attached through a gateway to network attached storage (NAS), FC, or iSCSI devices. Today and over the next five years, IB provides the highest bandwidth and lowest latency option. However, it does require a different fabric manager than used for FC or Ethernet, which is fine if the storage is all natively attached (e.g. Greenfield data center). However, for scenarios where IB is being added to an existing data center that uses FC or Ethernet for storage, IB increases the management expense.
4. IB based Fabric Convergence
Figure 5 depicts NAS and iSCSI based fabric convergence. This option uses Ethernet based adapters to connect servers and storage to an Ethernet fabric. The storage can be accessed through block (e.g. iSCSI) or file (NAS) mode IO and attached directly to Ethernet, for example IBM's N Series. NAS and iSCSI are faring well in mid-sized environments (e.g. remote office) and also in the middle-tier of multi-tier server environments.
5. NAS or iSCSI based Fabric Convergence
Today, a single Ethernet adapter can be used to merge storage and non-storage traffic onto the same link. However, Ethernet doesn't provide traffic differentiation capabilities. As a result, to provide service differentiation between storage and non-storage traffic an expensive IP layer solution is required. Traffic differentiation at the IP layer also adds per-hop latencies, which can be problematic for latency sensitive traffic, such as cluster Inter-Process Communications (IPC). Current Ethernet doesn't provide a low latency (layer 2) traffic differentiation solution, so using it to convergence cluster traffic with NAS or iSCSI may not work well for latency-sensitive environments.
For environments where latency is not a big concern, today's Ethernet can be used to converge non-storage and storage traffic. One example of such an environment is a mid-sized data center that aims to reduce management costs by using a single Ethernet-based network. If the environment is merging sockets traffic with storage (iSCSI or NAS) traffic and low-latency is not a requirement, then current Ethernet technology is adequate. Another example of where current Ethernet can be used is the middle-tier of a multi-tier server environment, which doesn't have low-latency requirements for communications between tiers.
A key difference between IB-based convergence and iSCSI/NAS based convergence is that, for the latter, only one fabric manager is required. An all iSCSI or NAS scenario reduces the management and hardware costs.
The Ethernet switch shown in Figure 5 to attach the storage can also be an iSCSI to FC gateway (a.k.a. multi-protocol switches). For this scenario, iSCSI attached storage, Ethernet-based adapters are used to connect the servers and storage gateways to connect the storage via FC. IBM's 2109M48 or Cisco's MDS 9216A devices are examples of these FC gateways. For scenarios where FC is already deployed, NAS and iSCSI obviously require a different network manager and the gateway to FC also needs to be managed. However, if the servers converge on an Ethernet link for storage (e.g. iSCSI) and LAN, then the hardware cost is lowered on the server.
The industry is working on enhancements to Ethernet that will benefit the iSCSI/NAS option and enable one more option. Part 2 of this article will cover the benefits of converged enhanced Ethernet.