System Design

Designing the
Ultimate Network Computer Board

Sun addresses the challenge of building a high-performance server with off-the-shelf components.

by Robert Feretich and Richard A. Raffel

The challenge for the design team at Sun Microelectronics (Sunnyvale, CA), a division of Sun Microsystems, was to capture a high-performance server on a PC-ATX form-factor motherboard. The division, which designed the 250-MHz UltraSPARC-II processor, applied its experience in modeling high-speed microprocessors to create the low-cost open network computing Ultra AX motherboard. A major design feat was the implementation of Sun's 83-MHz processor bus architecture using mainstream PC-board technology.

The Ultra AX board project was unlike any motherboard project ever undertaken at Sun. The design team was to deliver a motherboard that combined a high-performance processor-to-memory bus for server-class I/O with support for key PC architectural features and packaging standards to reduce system costs.

PC components would be used in the design at all levels of packaging except where there were technological arguments that Sun's internal technology was superior. Sun's internal CPU and system logic building blocks were used to boost the board's computing and processor-to-memory bus performance. Cross-bar memory switch building blocks from previous Ultra motherboard designs were used to give the Ultra AX board support for overlapping memory access. Both the processor and PCI I/O streams can simultaneously access the bus.

The use of off-the-shelf PC components sped the design of the EIDE and PCI buses. The selection of 72-bit, 168-pin dual inline memory modules (DIMMS) for the main memory enabled the design team to hit the memory density target of 512 Mbytes while minimizing memory costs to end-users. The Ultra AX board was also designed to allow integrators to take advantage of standard PC keyboards, PC-ATX enclosures, and PC power supplies.

The biggest design change from previous-generation Sun computing architectures was the adoption of the peripheral component interface (PCI) bus as the I/O bus for system add-in cards rather than the SBus. Part of the design challenge was interfacing Sun's Ultra Port Architecture (UPA) to the PCI bus. Fortunately, a Sun advanced development team was experimenting with PCI I/O adapters and had created a high-performance bridge ASIC that interfaced the UltraSPARC UPA Bus to both the 33-MHz and 66-MHz PCI buses.

With the incorporation of a 64-bit, 66-MHz PCI bus slot on a version of the Ultra AX motherboard, Sun is the first to implement the enhanced PCI (EPCI) bus. Initially, the wiring for the PCI bus section was done with an autorouter, followed by MOTIVE timing analysis from Viewlogic's Quad Design Technology (Camarillo, CA). Many of the routes were not optimal from a timing standpoint--the biggest problems being verifying clock routing and interrupt timing. The PCB designers did a manual iteration on the wiring to shorten some of the lines to meet the more difficult timing requirements. The wide range of PCI components is both a blessing and a curse. From the perspective of completing the Ultra AX board design, the availability of PCI peripherals and ASICs made it easy to implement the needed peripheral controllers. Crafting a PCI bus and a set of drivers that supports the entire universe of available PCI boards is another matter. Sun established a verification lab and a partners program early in the design cycle to ensure interoperability with a wide range of PCI boards.

Risk factors The complete board product required a complement of PCI device-driver software, firmware, and diagnostic software. We structured the Ultra AX motherboard project plan to identify and reduce risk as early in the schedule as possible. Delays in developing software were a key risk factor in the critical path of the project. As with many projects that involve hardware and software co-development, the challenge is to enable programmers to begin work before the prototype hardware becomes available. To reduce this risk, the previous generation of systems and experimental hardware were modified to yield a programmer's model similar to the expected Ultra AX motherboard.

Sun also built prototype PCI cards that contained building blocks of the various PCI logic. This same logic would later be integrated into the motherboard. The prototype PCI cards could also be used during the bring-up of the Ultra AX motherboard to provide a "workaround" hardware platform in case a particular peripheral in the prototype was flawed. The use of new and off-the-shelf building blocks helped reduce project risk because the blocks could be verified before they were integrated into the Ultra AX board.

The plans to build special debug hardware platforms allowed software and firmware development to start earlier and proceed independently of the Ultra AX board hardware design effort. Using this approach, we had high confidence in the basic hardware functionality, and completed a considerable amount of software debugging and testing before the first Ultra AX prototype board powered on.

Figure 1. With the UltraAX, 250-MHz RISC technology lands in a PC form factor enclosure with full support for the PCI bus and EIDE peripherals. The key design challenge was modeling and laying-out the 83-MHz zone between the processor, the UPA bus, and the crossbar switches.

Very wide, very fast With the major board features selected, we developed the finer details of the Ultra AX board's architecture. The fastest part of the board is the processor-to-memory interface, which is based on Sun's Ultra Port Architecture (UPA). The design team implemented the UPA bus functions using ASICs from groups within Sun, resulting in faster time to market. The UltraSPARC Processor System Controller and three-port crossbar memory switch building blocks came from prior Ultra board designs.

The UltraSPARC processor has incredible data movement capacity. The philosophy of moving large amounts of data quickly through the system was carried forward throughout the board design. Very wide (256-bit data with 32-bit error correction) memory busses were needed to feed the data-hungry UltraSPARC processor. The challenge of implementing the processor's UPA bus with 83 MHz clock rates using personal computer board design rules put Sun's design modeling and timing simulation expertise to the test.

Anchors in the floorplan The board's initial floorplan was driven by the location of the four PCI bus slots that protrude from the back left of the enclosure as well as the need to locate the Ethernet, keyboard, mouse, serial and parallel port connectors across the back right edge. The decision to support a Sun Creator 3-D graphics card (UPA 64-bit Slave) with its monitor connector position at a fixed location on the upper tier of the ATX I/O back panel was a key spatial constraint in the board's layout.

The Creator 3-D graphics card mounts horizontally above the motherboard, and it restricts the height of the components mounted under the card to approximately one-half an inch. Because of these claims by the interfaces for real estate, the connectors and CPU module boards dictated where the rest of the high-speed components were placed. The crossbar switches and the DRAM

DIMM sockets were pushed into a relatively small area between the UPA slave connector and the front of the motherboard (see Figure 1). For cooling purposes, the CPU was located near the fan at the back right corner.

Ultra AX competes in PC server space

The Sparc engine Ultra AX motherboard from Sun Microelectronics features the latest 167- and 250-MHz UltraSPARC processors (9.7 SPECint95 and 12.0 SPECfp95). The Ultra AX board is designed for network server applications in both LANs and telecommunications with on-board 10/100 Mbit per second fast ethernet and 400 kbit per second serial ports. The Ultra AX board has an upgradeable UltraSPARC processor module based on a standard Ultra Port Architecture (UPA) master port. The 64-bit UltraSPARC processor incorporates a fast floating point unit that supports Sun's Visual Instruction Set (VIS) extensions that accelerate audio, graphics, imaging, and video applications. With support for VIS instructions, processor block memory access is accelerated to speed up TCP/IP engagement. The Ultra AX board also has a UPA 64-bit slave socket for direct processor access to Sun's Creator 3-D 64-bit graphics card. The Creator 3-D graphic card is capable of rendering 2.4 million 3-D vectors per second and is supported by design packages such as Autocad from AutoDesk Inc. (San Rafael, CA) and various EDA tools from Cadence Design Systems Inc. (San Jose, CA). The Ultra AX board is the first Sun motherboard to support the PC industry's PCI bus standard. It has four PCI bus slots that support the 32-bit, 33-MHz PCI bus standard (two can operate in enhanced 64-bit, 66-MHz PCI bus mode). System builders can add a range of low-cost PCI cards including ATM and FDDI adapters, multimedia cards, RAID controllers, T1/E1 connectivity, SCSI adapters, and video cards. The Ultra AX board also has enhanced IDE (EIDE) connectors for hard disks and CD-ROMs.

Routing the 288-bit data bus at the intersection of the crossbar switches and the memory banks proved to be impossible with the initial number of interconnect layers in the printed circuit board. More layers were added to ease the congestion, resulting in the final stack up of six signal planes and four power planes. The layers and major wiring rules used in the layout of the Ultra AX board are shown in Figure 2. These rules are a good cross between Sun's internal board-rule set and those used by the PC industry for high-speed board design. Sun adopted board feature sizes, such as 5 mil. lines and spaces on internal layers, so a majority of PC board assembly vendors can manufacture the boards. In contrast, the technology used in some Sun workstation motherboards runs upwards of 14 layers, with much finer traces and pad placements allowed.

Sun chose middle-of-the-road design rules to enable proliferation of Ultra AX motherboard-type technology by multiple board vendors. Not only is the Ultra AX board a product in its own right, it is a demonstration of Sun's open technology for OEMs that want to create their own PC form factor board configurations using Sun Microelectronics' processor technology and ASIC chip sets.

The 83-MHz zone Beyond the initial placement, high-speed design requirements drove the physical design of the Ultra AX board. The decision to implement the UltraSPARC UPA Bus as a non-terminated bus operating at 83 MHz made component placement and wire routing extremely critical. The routing between the processor module connector, the system logic chips, and the crossbar switches was difficult. The routing required a significant amount of manual effort.

Figure 2. The board design rules (5 million lines and centers on inner layers) and technology selected for the UltraAX board can be easily manufactured by a majority of high-speed PC motherboard foundries.

The components on the UPA bus are connected by very wide busses and high-speed clocks, shown as the 83-MHz zone in Figure 1. The 64-bit, 66 MHz PCI I/O bus also challenged the
board designers. Sun focused on this section and modeled it with great care using Spice and MOTIVE. To perform accurate transmission-line analysis, we selected HSpice from Meta-Software (Campbell, CA) as the tool to perform electrical modeling.

Before starting analysis, we collected and audited the HSpice I/O models for the chips on the PCI and UPA busses. The decision to audit the models resulted from an observation by an engineer that the parameters on one of the driver models looked unusual. The results of the audits were surprising. The driver models that raised suspicions were in fact valid. Other models, which by initial appearance looked good, were found to be crude approximations. This audit saved the project time by not applying an accurate and rigorous analysis to invalid model inputs.

The most common problems found in the audit included the following:

• Some models were early approximations, and either a newer or more accurate model was available.
• Some models assumed I/O, VCC, and ground pin proximities that were not accurate for the pins that were being described.
• Inconsistent methods were used in accounting for the coupled noise from adjacent I/O signals.
• Some of the chip packages assumed in the I/O model did not match the package used on the Ultra AX motherboard.

Sun performed Spice transmission-line analysis of the non-terminated UPA bus. Simulation determined the actual launch and arrival times of critical signals and reflections on the bus. This analysis resulted in the generation of accurate timing data for the delay of each net under the actual loading conditions. Spice simulation takes into account the distributed RC tree conditions of each critical net as it is driven. Using this timing data, Sun calculated accurate minimum and maximum arrival times. Sun used traditional static timing analysis to verify the arrival times, signal timing margins, and clock-skew violations.

Sun performed the Spice modeling in two steps. In the first step, they adjusted the component data sheet setup and hold times to deduct the effects of the I/O drivers. Sun constructed a Spice model of the test conditions specified in the data sheet. Then, they simulated the network with the Spice model and determined the time difference between the driver input and output. This technique created a new table of setup and hold times for I/O driver inputs rather than outputs.

In the second step of Spice simulation, Sun used a spreadsheet to gather data on the critical paths in the design. The rows of the spreadsheet contained a list of the selected critical paths that needed to be measured accurately. The columns of the spreadsheet contained the following set of timing parameters for each net being analyzed:

• The adjusted Clock-to-Q delay and propagation delay values for the driving element.
• The duration between the launch and capture clocks.
• The clock skew.
• The receiving element's setup and hold times.
• The Spice simulation result for the actual wiring delay under the particular switching conditions.
• Computed timing margin or slack.

Since the UPA bus' nets were modeled as transmission lines, each possible source and destination pin pair on the critical busses was entered in the spreadsheet as a separate row to allow for the different instances of switching on the system buses. Multiple sheets were created and multiple simulations were performed to reflect different best-case and worst-case scenarios for the board.

Due to the combination of wiring analysis and Spice timing results, there were several iterations of the Ultra AX board's floorplan. Schematic capture, floorplanning, and Spice circuit simulation were performed concurrently until all timing conditions were satisfied. Then the completed Ultra AX board wiring design was sent for fabrication.

Bringing up baby The hardware and software development efforts were converging upon the board power-on date. The PCI software development started on standard PCs and migrated to the modified, advanced development systems. The software team was ready for the real board. Contingency hardware subsystems were ready in case serious problems existed on the Ultra AX board prototype. Faults would be isolated to a specific peripheral function, and work-arounds were ready to allow continued testing while bypassing the problem.

The lead board engineer traveled to the board vendor's assembly site and baby-sat the board as the vendor tuned its process and built the first batch. The first prototype board arrived at the lab at 5 p.m. on a Friday. The lab had been prepared with the necessary test equipment, spare parts, munchies, and high-caffeine beverages. The bring-up procedure had been worked out in advance and the initial team settled in and started an intensive visual inspection of the boards.

It is Sun's culture that "bring up" periods are periods of intense focus and dedication. The expectation laid on the team is that this period, starting with the arrival of the hardware and ending with the declaration of "Hello World," is a time of complete dedication by the project team. "Hello World" is defined as a board powering-on, loading the operating system, starting the windowing system, and using the e-mail tools and network connection to broadcast an e-mail message declaring "Hello World" to the engineering population at Sun.

"Hello World" So that no one would miss out on the event and so that team members would be willing to leave and get some sleep, the team agreed that once the ability existed to declare "Hello World," the actual event would be delayed until all team members could be contacted, gathered into the lab, and "Hello World" could be declared by the entire team. Thirteen days after the board arrived, on March 28, 1996, at 10:13 a.m., the Ultra AX motherboard team was assembled in the lab, the champagne was ready to be poured, and a team member hit the enter key--broadcasting to the Sun community that the Ultra AX board was alive.

A tremendous amount of work remained to be done. The major functionality and timing goals had been met by the first board prototype, yet the design was not perfect. Several different types of tests had been structured to prove the design and its implementation. Design verification testing validated earlier timing analysis with measurements coming in relatively close to the HSpice calculations.

The completion of the first wave of testing triggered the next release of the board, which included fixes to all of the bugs detected during the test stages. Many improvements were added for both testing and manufacturing. Pad shapes and component orientations were adjusted to improve assembly yields. On the bottom layer, test points were added providing inline circuit test (ICT) access to all nets. A special edge connector was implemented to facilitate the JTAG and functional test process. The new board is being tested in the field at hundreds of beta sites. The project team is cautiously waiting and monitoring in anticipation of the first customer shipment.

Robert Feretich is director of platforms engineering at Sun Microelectronics (Sunnyvale, CA).

Rick Raffel is the hardware manager and project manager for the Ultra AX product.

To voice an opinion on this or any Integrated System Design article, please e-mail your message to michael@isdmag.com.

integrated system design  March 1997