Advanced CMOS drives 100 GbE towards mainstream markets

Cost-effective, energy-efficient 100 Gigabit Ethernet (100 GbE) links will soon become essential tools for data center and service provider networks, which are struggling to satisfy the global economy’s relentless hunger for more bandwidth. The evolution of the 100 GbE transceiver interfaces necessary to support this growth is expected to follow a path similar to that of the earlier Gigabit Ethernet and 10 GbE technologies. These were first implemented as large, power-hungry, niche market products that used the steady improvements in commercial CMOS technology to produce ICs required to migrate to more compact, low-power industry-standard transceiver modules required for mainstream applications.

The first critical step on this path will be to migrate the 100 GbE transceiver’s high-speed SerDes function from its present implementation in exotic silicon-germanium (SiGe) processes to innovative low-power designs that can be economically fabricated using mainstream commercial CMOS processes. The second step in this evolution is to improve the signal integrity of the SerDes so it can be moved out of the 100 GbE transceiver module and onto the line card. This evolution will require high-speed, low-power CMOS architectures that provide complete solutions for the physical layer ICs inside the transceiver module and on the line card. Once achieved, they will enable production of next generation 100 GbE systems that offer up to 10X higher port density while reducing transceiver power consumption by more than 50%.

The Global Bandwidth Explosion

The 100 Gbps Ethernet standard and the technologies that will support it are being developed in direct response to the seemingly endless growth in bandwidth demand which threatens to overtake the current capabilities of data centers, service providers and enterprise networks¹. Wireless infrastructures face the same challenge as customers increasingly rely on their mobile devices to deliver streaming multimedia². As server-to-switch links migrate from 1 GbE to 10 GbE, the switches and router links that they connect are being forced to move from 10 GbE to 100 GbE. In order to accommodate the rapidly-growing traffic within the tight confines of a data center or cloud compute facility, the capacity of the line cards within those switches and routers will have to increase from today’s 480 Gbps levels, to capacities as much as 3 Tb/s per Line card and 30-40 Tb/s per chassis in the next several years³.

This presents a challenge for early 100 GbE technologies whose form factors and power consumption make it difficult to put more than two ports on a typical line card. In particular, first- generation 100G transceiver modules are expected to consume around 20W, a power level which presents significant cooling issues. In addition, the form factor of this first-generation module imposes a limit on the line card port density.

The Road to 100 GbE

Ratified in June 2010 by the IEEE Standards Association Standards Board, IEEE Std 802.3ba-20104 defines 100G Ethernet’s electrical, logical and physical characteristics to include options for transmitting Ethernet frames at 100 gigabits per second over multiple 10 Gbps or 25 Gbps lanes via single-mode fiber, multi-mode fiber, or copper cabling⁴.

For fiber media, the standard defines 100GBASE-SR10, an implementation which uses 100G optical transceivers based on 10 optical lanes at 10 Gbps, as well as the 100GBASE-LR4/ER4 standards which are based on four 25 Gbps optical lanes. Due to the complexity of a module capable of supporting 10 fiber interfaces, it is expected that 4x25G solutions will become the most cost-effective, energy-efficient, and commercially-viable solutions as the technology and the market environment matures. The first instantiation of a 4x25G optical fiber-based 100G solution will use the CFP form factor, which will be discussed shortly.

To complement these network-side technologies, the IEEE has begun efforts to define a specification for a 100G backplane that will support four lanes of 25G traffic across a copper-based PCB backplane, as well as a next generation 100G optical physical medium dependent (PMD) standard. 

100 GbE Technology Evolution

Like previous Ethernet upgrade cycles, first generation 100 GbE products are being built using exotic technologies available to produce functional, if not ideal, solutions. This will be followed by products based on standard CMOS technologies developed to meet the performance, power, packaging, and pricing thresholds required for widespread commercial adoption.

Most experts feel that 100 GbE will follow closely in the footsteps of 10G Ethernet, which used SiGe technology to build early-entry products and then transitioned to commodity CMOS as the challenges of using the mainstream CMOS process were solved. While SiGe-based optical modules were instrumental in kick-starting the 10 GbE market, it took CMOS to enable products with cost and power levels that were acceptable for mass deployment.

Echoing 10 GbE’s evolution, the first generation of optical modules supporting the 100GBASE-LR4 standard are using the CFP form factor and rely on electronic components fabricated in SiGe technology. As shown in Figure 1, the CFP module takes a low-risk approach by using 10G SerDes links on its line card-side interface. The 10G SerDes connections are well within the capabilities of the I/O blocks on several FPGA product lines and the IP cores commonly available for today’s mature CMOS processes. The module uses a pair of 10:4/4:10 multiplexer/demultiplexer (gearbox) PHY ICs to bridge the 10X10 Gbps data streams on the line card and the 4X25 Gbps data streams on the optical fiber.

While the Gearbox devices are well within the capabilities of today’s mainstream SiGe processes, the ten 10G SerDes interfaces add significantly to their size and power consumption. In addition, the physical space required to connect each of the devices’ twenty serializer/deserializer (SerDes) lanes requires the larger CFP module form factor and adds to the cost and complexity of the line card’s PCB design. Due to the CFP module’s size (145 mm X 82 mm X 14 mm) and its projected power consumption of around 20W, these early line card solutions will typically be limited to 1-2 ports per line card.

100G Module Progression

CFP modules will play an important role in paving the way for mass-adoption of 100 GbE technologies, but they must be quickly followed by succeeding generations of more compact, more affordable and more power-efficient products that can achieve the line densities required for mainstream commercial applications. As shown in Figure 2, the next step in the 100G module’s evolution will be the CFP2 module with approximately half the width of CFP, and a target power consumption of roughly 8W. The present roadmap ends with the introduction of the CFP4 module, which is expected to further reduce power consumption to roughly 5W. The transition to CMOS-based semiconductors will be a key enabler of this roadmap.

100G from the Semiconductor Perspective

The role semiconductors will play in the evolution of 100G technology becomes clear when one considers rates the key functional blocks in an optical interface shown in Figure 3:

• O/E Assembly: Optoelectronic components such as photodiodes, lasers, modulators, filters, and waveguides, which provide an electro-optical interface to the fiber.

• Amplifier/Driver: A set of laser drivers and transimpedance amplifiers that provide an electrical interface to the optoelectronic components.

• Retimer/PHY: High speed electronics devices that serve as the physical-layer interface (PHY) for the optical interface. These devices recover the timing information from the incoming data and use that to regenerate and retime the data and interface them to the system ASICs on the line card.

Regardless of module type, the Retimer/PHY element can be fabricated in either SiGe or CMOS. Due to its complexity and size, any energy or cost savings that can be realized in this element has a large effect on the overall solution. These effects become evident in the following scenarios which detail the semiconductor functionality required in each type of optical module.

The first-generation CFP-based system shown in Figure 4 contains all three basic electronic functions within the module itself. In this case, the four 25Gbps signals from the ROSA are passed through the PHY’s clock and data recovery circuit (CDR) to extract stable data streams. The PHY also contains a 10:4 Mux/Demux (also known as a gearbox) which converts the four recovered 25Gbps data streams into an array of ten 10Gbps lanes that feed the ASIC on the line card which performs the Ethernet Media Access Control and Physical Coding Sublayer (MAC/PCS) functions.

The block diagram in Figure 5 illustrates the change in functional segmentation that will occur with the adoption of the more compact, lower-cost, and lower-power CFP2 optical module. In this configuration, the 10:4 MUX/ DMUX function is moved to the line card and only the Retimer functionality remains within the module. A 4:4 PHY in the optical module uses SerDes transceivers to connect the four 25G data streams to the SerDes transceivers of the line card’s 10:4 PHY. The 4:4 PHY devices, built using mainstream CMOS, reduces the number of high-speed channels between the optical module and the line card from 10 to 4 in each direction, enabling a more compact and energy-efficient design.