The move away from PCI to new interconnect architectures has been a rocky road for communication designers. A myriad of interconnect architectures are being pitched and developed, all featuring disparate architectures. And, with so many options, choosing the right architecture can be a tedious and difficult decision to make.
Of course in an ideal world, members of the design community would have a single interconnect they could roll across all of their products. But, in reality, the communication interconnect will remain fragmented for the next few years, making bridging and interoperability between interconnects a vital design concern going forward.
To address these issues, the StarFabric Trade Association and the PCI Express Advanced Switching Working Group are working to better facilitate operation between their respective interconnects. In this article, we'll look at the work both organizations are engaging in. We'll also show how a hybrid solution can be built that allows designers to turn to StarFabric now and evolve to PCI Express AS over time.
Putting it Together
During the summer of 2002, the StarFabric Trade Association and AS Working Group announced that they would work together to ensure the two technologies can be used together. The organization recognized the similarities between their specs, and in turn, want to provide a situation where designers could begin developing around the StarFabric architecture today and evolve to the PCI Express AS spec to handle higher-performance tasks in the future. As Table 1 points out, some key elements of both specs are already aligned.
Table 1: Elements of Alignment in the StarFabric and PCI Express AS Specs
The StarFabric and PCI Express AS are switched interconnect architectures optimized for point-to-point backplane and inter-chassis applications. Both of these architectures can provide backward compatibility to legacy PCI systems.
In addition, there are numerous fundamental commonalities. Both architectures use path, or source, based routing. This eliminates the need for look-up tables on every switch in the fabric. Here, a packet enters the fabric with the exact route specified in the header.
The architectures are protocol-agnostic and support QoS via a credit based flow control algorithm. The different traffic types can be prioritized, which is critical in communication equipment that supports voice and data traffic. Each type of traffic will only be sent across a link if there are sufficient credits for its own type of traffic.
Both support peer-to-peer computing. In PCI Express Base and in StarFabric legacy mode, only a single processor and a single memory space are supported. In AS and StarFabric gateway mode, multiple processors are supported each with their own independent memory spaces.
Other commonalities include: hardware-based enumeration, CRC, hardware- based automatic failover, multicast routing and a layer software model. The latter will be discussed below.
Understanding the Differences
While there are many similarities, there are of course some differences between the specs. The biggest lies in the physical layer.
StarFabric PHY uses four 622-Mbit/s differential pairs to create a 2.5-Gbit/s links. The AS spec, on the other hand, calls for the use of four 2.5-Gbit/s differential pairs to create a 10-Gbit/s link.
The AS technology can be thought of essentially as an encapsulation technology. It can be used to transport and route any protocol by simply appending a small header with the appropriate routing information. The packet traverses through the AS fabric, and then, the header is stripped at the destination. In this area, AS is a simpler protocol than StarFabric.
A Hybrid Approach
The StarFabric spec and the AS spec can be seen as targeting different communication box designs: StarFabric in the access side predominantly and PCI Express AS in the higher speed metro and core applications that require 10-Gbit/s data rates and beyond. Companies in the industry are also pitching a hybrid mechanism that will allow StarFabric-enabled products to effectively run over AS backplanes through bridging functions.
Designers can also look to use StarFabric and AS in a complimentary manner in a system design. For example, Figure 1 shows a hybrid StarFabric/AS architecture. In this architecture, StarFabric technology could be included on the line cards to connect them to centralized AS-enabled resource cards containing DSPs or network processors. The StarFabric cards talk to the AS cards via an adaptive bridge. This bridge will have a number of 2.5-Gbit/s StarFabric links on one side and 10-Gbit/s AS links on the other. By implementing this approach, the line cards can provide low cost 2.5 Gbit/s connections while the AS switch card delivers the 10-Gbit/s bandwidth, which can provide the most optimal power-efficient and cost-effective, communication design.
Figure 1: Communication system featuring a hybrid StarFrabric/AS architecture.
There are two types of protocol inter-working supported in an adaptive bridge: tunneling and translation. AS is essentially a protocol-agnostic encapsulation technology. Any protocol can be assigned a protocol encapsulation interface (PEI), of which there are 256. PEI 0 through 7 are reserved for fabric management functions. PEI 8 through 223 are standard protocols. PEI 224 through 254 are user-definable.
Figure 2 illustrates how a StarFabric packet can get encapsulated into an AS fabric. In any tunneling system, the packets entering the fabric will exit the fabric unmodified. Thus, Ethernet packets stay Ethernet packets and PCI frames remain PCI frames. Using tunneling, system designers can take StarFabric designs to new levels of scalability.
Figure 2: Under the tunneling method, packets can move from a StarFabric-enabled card through an AS switch fabric card and back out untouched.
Now let's look at the translation bridging strategy. Figure 3 shows an embedded distributed processing system using both StarFabric and AS. Shown are three StarFabric-based embedded distributed processing systems, interconnected over an AS fabric. Also, attached to the AS fabric could be higher performance, shared resource cards.
Figure 3: Example embedded distributed processing system using both StarFabric and AS.
In the hybrid system shown in Figure 3, the translation function of the adaptive bridge is employed. In this system, each StarFabric-based subsystem would aggregate the intra-system traffic into AS fat pipes for connection to the high-speed AS-connected resources. Since the translation has a one-for-one functionality mapping between AS and SF, this function is straightforward and transparent to the applications. Note: Keep in mind that that AS/StarFabric hybrid systems can use both tunneling and translation simultaneously.
Making PCI Support a Reality
The vision and early design goals of both StarFabric and PCI Express resulted in a tremendous strength in their support for pure PCI legacy capability. In PCI Express Base and in StarFabric's legacy mode, existing PCI hardware and PCI software can be utilized without any modifications. No changes to operating systems, BIOS, bus enumeration code, or device drivers are required.
Software support, however, is critical to advanced features like high-availability, multicast, and quality-of-service in a switched interconnect implementation. While much is performed in hardware, the software infrastructure performs such functions as:
- Fabric discovery and enumeration
- Device mapping
- Path set up
- Interrupt handling
- Exception and error handling
StarFabric and AS have advanced features that make them ideal for communication and other embedded systems. To take advantage of these features, new StarFabric software utilities have been written. In addition, both the StarFabric and AS committees have defined a layered software model (Figure 4)
Figure 4: Promoting software interoperability, the AS and StarFabric specs are both developed around a layered software model.
The layered approach abstracts the details of the hardware from the higher-layer software. Thus, higher-layer software written for StarFabric today is re-useable when migrating to 10 Gbit/s AS in the future. If an algorithm is written below the hardware abstraction layer, the algorithm can be re-utilized, yet the code will have be modified to map the AS register level details.
Standardized Platform implementations
In addition to easing the software development process, industry members are also working with popular standards bodies to develop standardized platforms that support the migration between StarFabric and PCI Express AS. The PICMG 3.x specification is one of the more well known in the communication space.
The PCI Industrial Computer Manufacturers Group has recently approved its 3.x specification, which defined a new communication architecture, called AdvacedTCA, for communication designs. AdvancedTCA is based on switch-fabric architecture. However, the core specification is switch fabric-agnostic and details the mechanical, thermal, power and system management requirements common across technologies.
Underneath the PICMG 3.x effort, working groups have been setup to define backplane architectures based on the StarFabric and AS specifications. Thus, the AdvancedTCA architecture could make it even easier for designers to use 2.5-Gbit/s StarFabric Technology today and then evolve overtime to 10-Gbit/s AS backplanes.
Figure 5 shows such a system migration. On the left side, the system is populated with StarFabric node cards and two StarFabric switch cards. Each of the node slots is connected to four switch cards slots, two StarFabric and two AS. As the system requirements increase, end-users could add two AS switch cards and any number of higher performance AS node cards. They would be able to pay for just the amount of performance they required. The chassis would not have to be replaced.
Figure 5: Migration from StarFabric to AS in an AdvancedTCA architecture.
Commonly, in communication systems, service providers will install a shelf, and, once a threshold of usage is met, a second shelf is added. The shelves would be connected with an inter-chassis interconnect technology. The requirement is for the system to scale while still looking like one entity to the management systems. At some timeframe, they could migrate to a new generation platform. In this case, the current generation of chassis could be StarFabric-based. When AS is available, the next chassis could be AS-based. This hybrid StarFabric/AS system would meet the requirement and look like a single entity to the management systems.
StarFabric and PCI Express Advanced Switching share many commonalities that make them both ideal for next generation communication systems. Given these similarities, designers can start today with StarFabric, and when they need 10-Gbit/s performance, incorporate AS. They can do this all the while protecting the investment in StarFabric hardware and software.
About the Author
Tom Tinory is the director of application engineering of StarGen. He holds an MSEE from Worcester Polytechnic Institute and can be reached at firstname.lastname@example.org.