For the past few years, there's been a great deal of hype in the networking sector about moving away from custom IC designs toward off-the-shelf packet processing engines. This move, however, is full of challenges; and the biggest lies in the I/O.
When developers produced their own packet processing elements, it was easy to build custom I/Os to link them together. Unfortunately, that's not the case as developers move to outside network processors, traffic managers and more. Interfaces on these chips vary and are proprietary, forcing designers to buy full chipsets from one vendor or to turn to glue logic solutions that take up valuable real estate and power in board designs. Safe to say, these are not pleasant options for OEMs.
The Network Processing Forum is offering up an answer through its common switch interface (CSIX) specification. Through this spec, the NPF is hoping to create a common I/O that enables the selection of best-in-breed components and messy glue logic.
With the CSIX spec starting to mature, a host of chip developers are implementing this I/O on their chip designs. But to take full advantage of this I/O, designers need to full understand how it works. This paper will help out by looking at the CSIX L1 interface, which is the communication protocol definition between the traffic manager and the switch core. Specifically, this article will examine the architecture, unicast and multicast operation, frame formats, and more.
The CSIX L1 interface standardizes and defines the programming language between the traffic manager and the fabric, i.e. it characterizes how the traffic manager tells the fabric what to do with the data it is sending. This control information is passed in-band with data (ie. with the data and the control signals using the same paths) in order to combine the data and control information the traffic manager takes the data, such as IP packets, and breaks it down in smaller chunks. It then puts a header containing the control information over these data chunks to make a frame. These frames are called Cframes.
A Cframe can be up to 264 bytes, 256 maximum for payload and eight maximum for overhead. Cframes can be used to chop up data, such as the IP packets, into smaller units and route them through the switch fabric to the destination traffic manager. The Cframes will have sufficient information for the destination traffic manager to put together the original packet. Cframes can also be used to signal control information between the switch fabric and the traffic manager.
Among other things a Cframe header includes is the address of the destination traffic manager. CSIX allows up to 4096 traffic managers to talk to a single switch fabric.
Like most other frame structures, the Cframes are composed of a header, payload, and a trailer. The header features a variable length 2 to 6 bytes, the payload can be a maximum of 256 bytes, and the trailer is two bytes of vertical parity. Thus, the overhead in a CSIX architecture is 4 to 8 bytes per Cframe.
Figure 1 shows the typical structure of a CSIX Cframe. As this figure shows, the CSIX header is composed of a base header and an extension header. Every Cframe has a base header; the extension header is optional.
Figure 1: Typical structure of a CSIX Cframe.
The Base Header
Let's take a closer look at the base header and extension header. Figure 3 shows the base header. The base header is composed of 2 bytes. In Figure 3, bits 0 and 1 (R and P) of byte 0 are reserved for the base header.
Figure 2: Diagram of the CSIX base header.
In Figure 3, the 8 bits of payload length defines the maximum payload size of up to 256 bytes. The type field, on the other hand, indicates the Cframe type as in unicast/multicast, flow control, etc. Table 1 describes the type field encoding.
Table 1: Encoding for the Type Field
|Multicast Binary Copy
|Command and Status (Reserved)
In the base header, ready indicates if the transferring device (traffic manager or fabric) is ready to receive. There are two ready bits provided:
- Ready.0, which states that traffic manager is ready to receive unicast, multicast, or broadcast data.
- Ready.1, which states that traffic manager is ready to receive control information such as flow control.
The Extension Header
The extension header of the Cframe provides, among other things, the destination address and frame class (for class of service feature). The destination address tells which one of the 4096 traffic managers receives the frame. The fabric uses the 8-bits of class to provide the class-of-service (CoS). To illustrate hot this process works, let's look idle, unicast, multicast, and flow control Cframes
1. Idle Cframes: When there is no traffic, the links are kept synchronized by sending idle frames. This is nothing but idle traffic kept in place to keep links synchronized.
2. Unicast Cframes: The base header specifies that the frame is unicast and supplies the frame length. The extension header provides the 12-bit destination address (for 4096 traffic managers) and the 8-bit class.
3. Multicast Cframes: There are several forms of multicast addressing depending on the scheme used to choose the various list of destination traffic manager list. Here we'll look at two common multicast Cframes: multicast mask and multicast ID.
In the multicast mask Cframe, the extension header contains 8-bits [DDDD DDDD] of the traffic manager address and 16-bits of bitmap. Of the 16 traffic managers in the address range DDDD DDDD 0000 through DDDD DDDD 1111, the Cframe is sent to the bit positions set to 1 in the 16-bit bitmap. Using this method, up to 16 consecutive traffic managers can be addressed.
In the multicast ID method, the extension header contains a 22-bit tag. The fabric looks up and determines the set of destination based on the membership.
4. Flow Control Cframes: The CSIX fabric is allowed to exert two kinds of flow control using the Cframe:
- Link level flow control as specified by ready bits in the base header
- Flow control using the special Cframes.
The fabric in not allowed to ignore the link level flow control. The flow control can also be exercised by special flow control Cframes.
A single flow control Cframe can contain several of the flow control entries. Each of these entries tells the recipient the speed at which to send the traffic for the given destination, class and port. The fabric is allowed to ignore the flow control Cframe.
Signal Names and Pin Outs
CSIX traffic is a de-multiplexed 32 x n bit wide parallel bus where n=1 to 4. There are three control signals: clock, parity, and start of frame for each group of 32-bits.
In total, there are 35 x n lines in each direction, thus a single 32-bit wide CSIX bus would occupy 70 pins while a 128-bit CSIX would bus would occupy 280 pins.
Figure 3 shows a 32-bit CSIX bus. All signal references in the specification are with respect to the switch. Thus, the Rx portion of Figure 3 represents the signal received by the switch while Tx portion represents the signal sent out by switch. The control signals are driven by the same entity that drives the data paths.
Figure 3: Pinouts for a 32-bit CSIX bus.
CSIX can be implemented using one of the three logic levels as shown in the Table 2.
CSIX Signal Classes
Real Design Examples
Now that we've laid out the basic elements of the CSIX specification, let's explore a few real-life examples. In example 1, let's explore situation where a 1024-byte payload from line card A must to be transferred to line card N. Line card A has already stripped the packet and using the header information it has determined that the packet is of priority level 2 and the destination port is line card N. Furthermore, the system designer has decided that the CSIX packet size is to be 80 bytes.
Once the above conditions are set, the network processor on line card A will then segment the packet into 80-byte Cframes as shown here:
Original packet:Click here to view Original Packet
An 80-byte Cframe will have 74-byte payloads as shown:Click here to view the 80-byte Cframe
The network processor will generate a host of idle Cframes. Assuming that there is no traffic from the traffic manager to the crossbar, a constant flow of idle Cframes will move between the line card and the crossbar to keep the link alive. These Cframes have the following encoding: C0 0P p
Now a base header is sent with ready bits that onfo the fabric that the traffic manager is ready for data. This base header then informs the traffic manager the fabric is ready for flow control. In this example, we'll assume that the traffic manager is ready for both data and flow control. Therefore, bits 6 and 7 of byte 0 are 00. Type idle, 00, CSIX reserved bits are being set to 0. It's also important to note in this sequence that there is no extension header, that the payload is 0 and the frame checking sequence (FCS) is pp.
Assuming that there is no traffic before the dispatch from above, the line card will continually send idle Cframes. For the purpose of our example, however, we will send out an Ethernet packet header, dubbed Cframe1, over the CSIX bus. Cframe 1 is represented as:
Cframe1:Click here to view Cframe1
In addition to housing the Ethernet packet header, Cframe 1 features the destination address and is designated a a unicast Cframe with a class of 0.
Once Cframe 1 is received, a base header is then sent out from the traffic manager over the CSIX interface. This header again tells fabric that the traffic manager is ready for data. It then tells the fabric that the traffic manager is ready for flow control. We are assuming that the traffic manager is ready for both data and flow control. Therefore, bits 6 and 7 of byte 0 are 00. Type = unicast, 0x1, CSIX reserved bits are being set to 0.
Unlike the idle packets above, we now have an extension header that is 4 bytes, set to class 0, and offers a destination address of N. We also have a payload of 72 bytes. FCS is still set at pp.
Cframe2 (see below) is now sent out over the CSIX bus. This Cframe is the continuation of the ethernet packet. It contains the same information as the previous packet, as described below.
Cframe2:Click here to view Cframe2
Once the Cframe 2 is sent, a base header is also sent over the CSIX interface. This base header performs the same functions as above. In this case the extension header and payload remain the same but the FCS is now set to acdb.
Now Cframe3 is sent out over the bus. Cframe 3 is also a continuation of the Ethernet frame. This Cframe, and all that follow, are handled using the same method as above same method until all frames of the Ethernet packet have been sent. No padding is needed, as CSIX is a variable frame interface, and the fabric will take care of all padding.
Cframe3:Click here to view Cframe3
In the first example, we showed a single line card send info to another line card. Now, we'll show line card A and line card B sending frames to line card N.
To start the transmission, line card N sends a flow control signal to line card A. The flow of Cframes between line card is the same but the destination is blocked. Therefore, line card N will send flow control frame to the swtich fabric and depending upon the class of data, the fabric will send the flow control to either line card A or B. It will do so by overriding the flow control information on the idle frame. This idle frame will be encoded as: 40 0 p p.
The idle Cframe indicates that the destination traffic manager is busy and can not accept any traffic. The flow control information is static in the sense that this destination traffic manager is considered to be blocked until it sends a flow control frame indicating that it is now available to receive traffic.
More Info on CSIX
CSIX is quickly becoming a de-facto standard supported by various vendors. More information on CSIX can be found at www.npforum.org.
About the Authors
Tobi Potash is a field application engineer at Vitesse. Tobi has a BSEE from California State University, Northridge and is completing a MBA at the Anderson School of Management at UCLA. Tobi can be reached at email@example.com.
Raj Budhabhatti is a field application engineer at Vitesse. Raj has an MSEE degree from Louisiana State University and can be reached at firstname.lastname@example.org.