Backplane tutorial: RapidIO, PCIe and Ethernet

The trend towards leveraging a common SerDes technology will continue with future versions of these specifications. The implication is that raw bandwidth is not a significant differentiator for these protocols. Instead, the usefulness of each protocol is determined by how the bandwidth is used.

Protocol summaries
Most designers are familiar with basic Ethernet protocol characteristics. Ethernet is a 'best effort' means of delivering packets. The software protocols built on top of the Ethernet physical layer, such as TCP/IP, are necessary to provide reliable delivery of information, as Ethernet-based systems generally perform flow control at the network layer, not the physical layer. Typically, the bandwidth of Ethernet-based systems is over-provisioned by between 20 and 70%. Ethernet is best suited for high latency inter-chassis applications or on-board/inter-board applications where bandwidth requirements are low.

PCI Express (PCIe), in contrast, is optimized for reliable delivery of packets within an on-board interconnect where latencies are typically in the microsecond range. The PCIe protocol exchanges Transaction Layer Packets (TLPs) such as reads and writes, and smaller quantities of link-specific information called Data Link Layer Packets (DLLPs). DLLPs are used for link management functions, including physical layer flow control. PCIe was designed to be backwards compatible with the legacy of PCI and PCI-X devices, which assumed that the processor(s) sat at the top of a hierarchy of buses. This had the advantage of leveraging PCI-related software and hardware intellectual property. As discussed later in this article, the PCI bus legacy places significant constraints on the switched PCIe protocol.

RapidIO technology has been optimized for embedded systems, particularly those which require multiple processing elements to cooperate. Like PCIe, the RapidIO protocol exchanges packets and smaller quantities of link-specific information called control symbols. RapidIO has characteristics of both PCIe and Ethernet. For example, RapidIO provides both reliable and unreliable packet delivery mechanisms. RapidIO also has many unique capabilities which make it the optimal interconnect for on-board, inter-board, and short distance (<100 m) inter-chassis applications.

Physical layer
At the physical/link layer, the protocols have very different capabilities when it comes to flow control and error recovery. Ethernet flow control is primarily implemented in software at the network layer, as this is the most effective for large networks. Ethernet's only physical layer flow control mechanism is PAUSE, which halts transmission for a specified period of time. The limited physical layer flow control means that Ethernet networks discard packets to deal with congestion.

In contrast, PCIe and RapidIO physical-layer flow control mechanisms ensure reliable delivery of packets. Each packet is retained by the transmitter until it is acknowledged. If a transmission error is detected, a link maintenance protocol ensures that corrupted packets are retransmitted.

PCIe ensures delivery using Data Link Layer Packets (DLLPs), while RapidIO uses control symbols. Unlike DLLPs, RapidIO control symbols can be embedded within packets. This allows RapidIO flow control information, such as buffer occupancy levels, to be exchanged with low latency, allowing more packets to be sent sooner. Figure 1 illustrates this concept. In the leftmost panel, Device A cannot send any packets to Device B because the buffers in Device B are full. Device B is sending packets to Device A continually. In the middle panel, one buffer in Device B becomes free. At this point, Device B must inform Device A that it can send a packet. In the PCIe panel on the bottom right, the DLLP cannot be transmitted until transmission of the current packet is complete. In the RapidIO panel on the top right, a control symbol is embedded in a packet that is being transmitted. This allows the RapidIO protocol to reliably deliver packets with lower latency and higher throughput than the other protocols. The ability to embed control symbols within packets allows the rest of the RapidIO flow control story to be much richer than PCIe or Ethernet, as discussed later in this article.

(Click to enlarge)
Figure 1. RapidIO embedded control symbols and PCIe DLLPs. (See text for description.)

Beyond more efficient flow control, the ability to embed control symbols within packets gives RapidIO an ability that currently neither PCIe nor Ethernet can offer. Control symbols can be used to distribute events throughout a RapidIO system with low latency and low jitter, as shown in Figure 2. This enables applications such as distribution of a common real time clock signal to multiple endpoints, or a frame signal for antenna systems. It also can be used for signaling other system events, and for debugging within a multiprocessor system. As shown in Figure 2, PCIe DLLPs introduce a significant amount of latency and jitter every time the DLLP is transferred through a switch. In contrast, the RapidIO protocol allows a signal to be distributed throughout a RapidIO fabric with less than 10 Unit Interval (UI) of jitter and 50 nanoseconds of latency per switch, regardless of packet traffic.

Figure 2. RapidIO Multicast Event Control Symbol and PCIe DLLP.

PCIe and Ethernet may choose to extend their respective specifications in future to allow events to be distributed with low latency. Introduction of a control-symbol like concept would be a large step for Ethernet. Several initiatives are underway within the Ethernet ecosystem to improve Ethernet's abilities within storage applications that may need a control-symbol like concept. Ethernet is also being enhanced to incorporate simple XON/XOFF flow control.

PCIe currently does not allow DLLPs to be embedded within TLPs as this concept is incompatible with the legacy of the PCI/X bus operation. DLLPs embedded within TLPs create periods where no data is available to be placed on the legacy bus. PCIe endpoints could operate in store-and-forward mode to ensure that packets are completely received before being forwarded to the bus, at the cost of a drastic increase in latency and lowered throughput. Given the PCIe focus of on-board interconnect for a uniprocessor system, and the continued need to maintain compatibility with legacy bus standards, it is unlikely that the PCIe community will be motivated to allow DLLPs to be embedded within TLPs.