LIFO or FIFO? How to accurately measure data center Ethernet latency

A key performance metric in data center networking is latency. Because of this, high frequency financial trading (HFT), high-performance computing (HPC) and similar performance-sensitive environments, utilize ultra-low-latency 10 GbE switching technology to handle the huge volumes of network traffic that represent millions of transactions each day. The relentless increase in transaction data rates in electronic trading and the need to remain competitive are driving financial-market stakeholders to invest in the latest Ethernet switching technology to support spiraling bandwidth requirements and drive down transaction latency.

Consequently, latency measurements have now become an important part of data center Ethernet switch vendor test reports. When evaluating the underlying switch ICs for these systems, designers need to be aware that there are several methods for measuring latency, and that selecting a design based on the wrong methodology could be a mistake as some methods don't represent real world performance.

These measurement methods have recently been used in tests that have enabled some LAN switching vendors to claim ultra low latency.  How is this possible? Using the RFC 1242 last-bit-in, first-bit-out (LIFO) methodology, store and forward switches can actually report sub-microsecond latencies. RFC 1242 also states however, it's preference for a first-bit-in to first-bit-out (FIFO) approach: "ideally the measurements for all devices would be from the first actual bit of the frame after the preamble."  This more accurately accounts for any storage or processing time that packet encounters as it traverses the switch. 

Switch Architecture and Latency

The first low-latency 10GbE switch, introduced in 2006, achieved its latency by using a cut-through mode of operation where the switch may start transmission of a frame before the frame has been completely received. Store-and-forward switches, on the other hand, cannot achieve low latency since the packet must be completely stored in memory before being transmitted out the egress port. But not all cut-through switches are created the same when it comes to latency as the underlying switch architecture makes a big difference.

Memory access bandwidth has been a thorn in the side of switch chip architects. When using traditional cross bar memory designs, there is insufficient on-chip bandwidth to allow every input port to write into the same output queue simultaneously. To get around this blocking issue, chip architects may include virtual output queues at every switch input, known as a combined input/output queued (CIOQ) architecture (Figure 1).

Virtual output queues provide at each ingress port a single queue for each switch output (egress) port. If a particular egress queue is temporarily blocked, the matching ingress queue will be flow controlled, but packets destined for other egress ports can bypass this blocked queue and send data to other non-blocked egress ports. For an N-port switch, however, this means N*N input queues and associated schedulers, which adds significant complexity. It also adds to packet latency since each packet must be queued twice through the switch. Because of the complexity of VOQs and associated schedulers, many switch designs trade off complexity at the expense of some level of internal blocking, which further adds to latency.

New shared memory switch technologies utilizing performance advances in on-chip crossbar and memory design, provide the ability for switches to support a fully non-blocking output queued, shared memory architecture with extremely low latency, as shown in the Figure 2. By utilizing a proprietary high-bandwidth memory structure, the switch architecture can be simplified, eliminating the complexity of ingress VoQs and the extra memory they require. In addition, multicast packets are only stored once, further reducing on-chip memory requirements. This also provides the lowest latency with only a single memory queue/dequeue and very little internal blocking.

In order to evaluate these switch architectures, one can examine some of the recent reports from third party test houses that compare the latency of data center top of rack switches from several vendors. But, one must be careful when interpreting the results.

There are several ways to measure latency through a switch: first-bit-in to last-bit-out (FILO), last-bit-in to first-bit-out (LIFO), first-bit-in to first-bit-out (FIFO) and last-bit-in to last-bit-out (LILO). In each case, latency is measured at the switch ingress and egress ports.

FILO

This method measures the latency of the switch plus the time it takes a packet to be transmitted out the egress port interface. Although this may be adequate for measurements such as application-to-application latency in a server cluster, it is not a very good way to measure switch latency. To illustrate this point, consider an electrical connector. The FILO method would suggest that the connector has latency that depends on packet size, when in fact the connector has zero latency.

Large packets add significant latency to these measurement results, making it more difficult to extract the actual switch latency. Finally, consider two cut-through switches in series. Using the FILO method, the latency through two switches cannot be calculated by adding the measured latency of a single switch. Because of these factors, this is not a proper method to use for measuring switch latency.

LIFO

The LIFO method was used in the late 1980s and early 1990s to measure latency in telecom networks that transmitted fixed size cells. Some test reports still reference LIFO in RFC 1242, which was written in 1991, long before high bandwidth cut-through switching was developed. For vendors that only offer store and forward switches, this method of measurement can make their results look much better due to the fact that it does not take into account the time it takes for a packet to be completely stored in the switch before forwarding. As we will see below, cut-through switches cannot be accurately measured using this method.

FIFO / LILO

These methods are effectively the same and are the only way to properly measure the latency through a cut-through switch.  To demonstrate how these methods work, and how they compare to LIFO testing, let's evaluate some live test results based on recent testing using a Fulcrum "Monaco" 10GE switch.

FIFO vs. LIFO Actual Test Results

The Monaco reference platform contains a 10GbE switch chip that provides 24 SFP+ ports in a 1U form factor. The latency in this platform includes the latency through the switch plus the latency through the SFP+ PHYs. Figure 3 shows latency measurements using both the FIFO and LIFO methods when the switch is operating in cut-through mode.

As can be seen, the FIFO results show the actual cut-through latency of the switch while the LIFO numbers are artificially reduced by the time it takes to completely receive the frame. For a 10GbE link the relationship between these two measurement methods can be described in the following equation:

     LIFO latency = FIFO latency - (Frame Length + 20)*0.8 nS

Since the LIFO measurement method could result in negative latency in cut through switches like Monaco, the tester reports this as "zero" latency. This means that a comparison of cut-through switches for larger packet sizes will be meaningless, as they will all show zero latency.

Conclusions

Latency is becoming a key parameter when selecting networking equipment for the data center. Network designers may utilize test reports to compare switches from different vendors, but these reports must be examined carefully as there are several ways to measure latency and accurate comparisons can only be made if the latency of competing chips is measured using the same method. Cut-through switches provide the lowest latency networking solutions for the data center. This paper has shown that the only proper way to measure latency for cut-through switches is first-bit-in to first-bit-out.

About the Author

Gary Lee is director of product marketing for Fulcrum Microsystems and has been working in the semiconductor industry for over 29 years.  For the last 14 years he has been involved in the development of switch fabrics for the telecommunications, data communications and storage industries. While at Vitesse Semiconductor, he was a key member of the team that developed the CrossStream, GigaStream and TeraStream switch fabric families and holds patents in this area. He has also worked on ASI, PCI-Express, SAS and Ethernet switch fabrics. Gary has a BSEE and an MSEE from the University of Minnesota