The Compute Ranch Takes Over EDA

How are you going to keep them down on the mainframe, after they've seen the compute ranch? For the engineers on this spread, there's only one line--and no waiting.

by Dwayne Lee

Today, larger and more complex designs, increasing global competition, and narrowing product windows are challenging the ability of many companies to move functional products out the door in time to invest in the next development cycle. The ability to deliver on or before deadline often means the difference between success and failure for an engineering organization. The results of missing that product window, or shipping a defective product, can often prove catastrophic for a company.

The extremely competitive environment that characterizes today's electronic marketplace demands processes that yield functional product on the first manufacturing pass. For most organizations, the expense of testing physical prototypes is no longer an option. The key to functional first-silicon IC, circuit board design, or system design lies in extensive software testing and simulation. The more cycles the engineers are able to model in software before completing the design, the more robust the end product. The complexity of today's processors and designs precipitates the need for billions of simulated cycles within a typical product development cycle.

Network-centric compute ranches have emerged as an effective method for boosting a design team's productivity by delivering the compute muscle the designers so desperately need. Well-designed compute ranches provide a scalable computational resource and a seamless engineering environment that enable true 24-hour, seven-day-a-week global engineering. Especially ideal for electronic design automation (EDA) environments, the ranch brings large amounts of power to bear on the computationally intensive problems typical of EDA tasks, thereby increasing design efficiency.

Home on the ranch

Sun Microsystems has had considerable experience and success with compute-ranch technology, deploying several such installations in its design centers around the world. In fact, we have recently used large compute ranches to design our flagship products. Until just a few years ago, of course, the compute resources on the designer's desktop provided adequate might for most electronic design and analysis tasks. Today's designs, however, are making the desktop-centric model obsolete--even the currently available high-performance workstations can't handle the most demanding tasks. Consider our own microprocessor designs. In 1991, our processors averaged three million transistors; the number increased to five million in 1995 and today hovers at 22 million. The next generation of processor designs now threatens to push the count into the 50-million transistor range.

The ranch brings large amounts of power to bear on the computationally intensive problems typical of EDA tasks.

Along with this explosive growth in transistors, the speed of these processors rises exponentially as the physical geometries shrink. The designs themselves are getting more complicated, with seven layers of metal becoming commonplace in large-scale microprocessor designs. Clearly, running full simulation and verification suites on these increasingly sophisticated designs will require decades, if not centuries, of computing time to complete, even with the fastest work-stations available today to the design community.

Since product deadlines are seldom flexible, productivity must improve by orders of magnitude to ensure that a quality design meets its schedule. But most organizations have reached the point where adding engineers or upgrading designers' desktop computers--or even adding a few powerful network servers--won't begin to satisfy this endless appetite for increased efficiencies. How then to deliver the robust compute resources needed for fast and effective simulation? Clearly, this issue demands immediate attention.

Design organizations are realizing that raw computing power currently dispersed across disparate design groups within a company needs to become part of a storehouse of shared network resources. A network of computers can provide enormous computing capacity, often rivaling that of the most powerful supercomputers. Compute ranches combine large numbers of desktop and server systems into a single networked entity. From the designer's perspective, well-constructed compute ranches represent a seemingly inexhaustible computational resource. Rather than addressing individual systems, engineers submit jobs directly to the compute ranch, freeing their own desktop systems for other important design-related tasks.

As mentioned, we operate several compute ranches across our intranet. Though the configurations are constantly evolving, the EDA compute ranches used for our microprocessor design group generally include a formidable array of components: 750 multiprocessor UltraSparc systems, approximately 2,500 UltraSparc CPUs, over a terabyte of physical RAM, approximately 26 terabytes of disk space, and 100-Mbyte fast Ethernet switched networking throughout (see Figure 1).

Our typical ranch simulates over a billion cycles in a given week and can work simultaneously on upwards of 15 to 20 projects, all at different points in their design cycles. Such a configuration supports roughly 600 to 800 designers and as many as 200 different design and simulation applications, including approximately 50 mainstream EDA applications. By leveraging network computing, centralized administration, and significant automation, we can minimize the support staff--averaging just one administrator for every 100 systems.

Figure 1 - Ranch access
The physical layout of a large compute ranch provides the key to use and maintenance. Ease of access facilitates the swapping out of systems.

Deploying only the latest and fastest computing hardware makes the best resources available to each design team. Because the ranch provides the capacity for enormous compute-intensive jobs, the designer's desktop system no longer needs to incur the expense of the latest processors or the most memory and disk space. In fact, older systems are commonly rotated out of the ranch, serving subsequently as desktop systems for the organization even as the ranch gains newer hardware. Occasionally during off hours, the ranch may utilize idle cycles available on the older CPUs now residing on the engineers' desktops. This extra muscle can make a big difference when an IC design is nearing tape out and requires additional compute cycles for verification.

The care and feeding of servers

Creating a successful ranch demands more than simply attaching all available computer resources to the network. In fact, hastily assembling a compute ranch without proper forethought can lead to serious design inefficiencies. Unfortunately, the ad-hoc nature of the ranches found in some companies means that the designers themselves often manually launch, manually monitor, and--should there be a failure--manually restart simulation jobs. That process translates into valuable engineering time wasted, first trying to find available systems and then determining which system contains sufficient resources to handle the job. Additional problems arise if the system lacks access to the EDA licenses necessary for a particular task. Making the wrong choice here can and often does lead to job failure. When simulations take days or weeks to run, failed jobs translate into lost productivity, which in turn seriously threatens a team's ability to meet schedules and cost commitments. Clearly, planning both the layout and the administration of the ranch in advance of configuration delivers an optimized ranch for an organization.

Effective use of all of the resources within the ranch requires that the ranch hands take heterogeneity and the unique properties of the individual hosts into consideration. Sophisticated network computing techniques and significant amounts of automation become necessary to maximize valuable investments in critical components. In this environment, resource-sharing software becomes imperative to guarantee the efficient use of all available compute power.

Ideally, the software provides enough flexibility in its distribution algorithm to pool a large number and variety of resources and, therefore, an overall increase in throughput. The resource-sharing software we have developed internally asks users to provide the criteria for a specific job, including the priority of the job and the required EDA software. The software matches up the jobs submitted with the appropriate servers available, then schedules the user's job for execution. The resource-sharing software oversees fair-share execution of jobs--eventually, even low-priority jobs execute. In addition, we know that if a job starts, it finishes. In other words, a particular job won't run out of memory, disk space, swap space, or licenses during a simulation run.

An automated job submission process allows engineers to achieve considerable parallelism when launching many jobs simultaneously. The resource-sharing software tracks all of the jobs under its control and even restarts stalled jobs automatically in case of failure. In addition, users can monitor the status of jobs underway and are notified upon completion.

Figure 2 - Server loading
Load-sharing software prioritizes jobs and maximizes server utilization. Naming the myriad servers offers a rare opportunity for a little creative license.

Our load-sharing software provides multiple adjustable queues for job submission, each queue associated with a different priority level or different types of jobs. For instance, one queue might be designated for very short high-priority jobs in need of a quick turnaround response. Another job queue might handle extremely long jobs--such as test sets for full design verification--that typically generate large data files. Custom scripts interface with the load-sharing software to select and dispatch jobs based on resource requirements. The load-sharing software then schedules, prioritizes, and initiates each job on the appropriate host (see Figure 2). Upon completion, the software sends the user e-mail notification. The software also generates reports on the general status of each job.

A network of computers can provide enormous computing capacity, often rivaling that of the most powerful supercomputers.

This resource-sharing software has produced impressive results for us. The servers in our ranch commonly approach 90 to 100 percent utilization. Our designers, freed from having to chase failed jobs and track system administration issues, can do what they do best: design. Running large numbers of jobs gives designers access to more analysis data and identifies problems more quickly. Engineers using the ranches don't think twice about running extra tests--even those consisting of thousands of jobs--to more easily visualize trends and determine design parameters. One user recently averaged over 3,500 jobs per day, eventually utilizing over 25 years of compute time in a single month.

Compute ranches have proven especially powerful in running verification tests, especially when a massive design such as a microprocessor nears tape out and the verification task reaches equally mammoth proportions. One microprocessor design close to completion required up to 2 billion cycles of simulation a week. That requirement, shared simultaneously across several designs, defined the work load our ranch had to handle. In addition, today's processor geometries have shrunk to a point that the parasitic effects of the physical interconnects require full-chip verification and analysis. In fact, gate-level verification for the performance of the entire chip has evolved into a single seamless process as a result--an efficiency unthinkable prior to the advent of large, well-organized ranches.

Rounding up the tools

Compute ranches also offer some very specific strengths with regard to EDA tool usage, leading to a much more efficient and effective use of design applications and tools. Most ranches include a variety of specialized servers for specific tasks. For example, servers are designed to process computationally intensive jobs as quickly as possible. File servers provide project design data to the compute servers and offer storage for log files and data generated by a simulation. In addition, some specialized servers are dedicated to supporting EDA tools and licenses, because EDA software acts as a critical asset in helping a design team to create cutting-edge designs while meeting an aggressive schedule. These servers have helped us to optimally deploy valuable EDA resources across multiple design teams working on a variety of different projects.

We know that if a job starts, it finishes--a particular job won't run out of memory, disk space, swap space, or licenses during a simulation run.

Various applications or tool servers provide a centralized location for installed software packages and custom scripts. By standardizing on a supported tool set and centralizing its administration, application servers deliver greater tool availability and reliability. Sometimes, the ranch may contain project-specific application servers to house special tools that are unusually expensive or that demand extra compute power. For instance, we have dedicated eight servers to a particular physical verification tool, a tool crucial to the verification of our deep-submicron processors as they near tape out. To minimize the expense of this particular license, we bought as few seats as possible, and dedicated servers minimize the delay in accessing this important resource.

Because application servers house all of the critical applications for the ranch, we view them as a crucial resource. Our ranches therefore contain redundant application servers that enable load balancing among application servers and eliminate single points of failure.

To complement the application servers, we have also dedicated a set of servers solely to EDA licenses and attendant software. Software licenses constitute a significant portion of the cost of an EDA ranch. Not uncommonly, organizations spend as much money, if not more, on software licenses as they do on the ranch hardware. Obtaining sufficient quantities of the necessary licenses and using them as frequently as possible becomes essential to optimizing these costs. We didn't want too many licenses, but we certainly wanted enough to ensure maximum design efficiency. Possessing too few licenses can decrease productivity, while purchasing too many simply wastes financial resources.

Figure 3 - Ranch management
Reports generated by resource-sharing software help to track jobs queued, running, and finished. Hourly job usage can identify utilization peaks throughout a 24-hour period.

Today, most EDA tools offer distributed or floating licenses that any system in the compute ranch can check out. During the normal course of a day, there are always several machines that aren't functioning properly. Floating licenses allow the system to immediately reallocate the jobs to other machines, so that if a machine goes down, the ranch hasn't lost the use of the license. The license servers usually include the ability to support fail-over among several license servers. We deploy triple-redundant license servers to ensure availability of license services. Like application servers, license servers represent a critical resource that must be readily available to ensure uninterrupted operation of the ranch.

Meanwhile, for those tools that still maintain node-locked licenses that effectively bind an application to a particular system, we supply dedicated resources to support them. We try to avoid node-locked software as much as possible because it limits the effective system utilization in the ranch.

Table - Thousands of jobs a day
	Mar '99	Feb '99	Jan '99	Dec '98	Nov '98	Oct '98
Total jobs dispatched	630,865	586,203	680,998	826,331	865,928	957,975
Average queue time (hrs)	7.006	9.179	6.336	5.804	4.872	3.714
Average run time (hrs)	1.326	1.194	1.282	1.038	1.047	0.882
Total run time (hrs)	795,491	659,151	744,441	801,235	814,496	794,890
Utilization of general servers	98%	98%	96%	98%	96%	97%

The EDA benefits

Centralizing compute resource and employing application and license servers causes the ranch to yield significant benefits in terms of EDA resources. First, the techniques encourage designers across different projects to use identical tools--a way to leverage tool skills and experience between design teams. And as any seasoned designer knows, switching tools can prove to be a very time-consuming proposition. By using the same set of tools for similar designs, the team doesn't have to waste valuable time and money learning and installing new applications. Extending a methodology across several teams and derivative designs enables those teams to focus almost exclusively on design creation. Of course, new tools and capabilities will evolve into the mix as needed and as next-generation EDA innovations become available.

By standardizing on a supported tool set and centralizing its administration, application servers deliver greater tool availability and reliability.

Second, a compute ranch also minimizes tool redundancy, which makes the most efficient use of the licenses and tools already in hand. The dedicated application and license servers ensure that tool licenses enjoy as much use as possible within the strategy of the design process. The resource-sharing software dynamically controls the distribution of the EDA licenses based on project priority. The scheme allocates licenses where and when they make the most sense. Obviously, the compute ranch enables the optimal use of EDA tools, saving millions of dollars in EDA costs each year.

Third, a centralized resource database--gathering information on every event in the ranch--helps to minimize design bottlenecks and aids in forecasting future EDA needs. The resource-sharing software uses this data to generate reports that monitor the use of design and analysis applications. These reports compile valuable information on designer usage, revealing which groups constitute the biggest users and listing that usage of resources by project, queue, and user (see Figure 3). Subsequently the team managers can more easily understand the root cause of various user problems and to propose appropriate solutions.

These reports also provide guidance in projecting future ranch use and for determining future EDA purchases. To ascertain how many licenses we need, we carefully examine the detailed reports, quickly arriving at the ideal number of licenses that we should purchase. We continually track the usage, then update and revise our licenses to meet changing design requirements. In one case, we found that our designers were using one particular set of expensive licenses only at peak times; we immediately encouraged the designers to spread their usage of that particular license throughout the day. In another case, after analyzing the reports, we decided to purchase more copies of a heavily utilized license.

Taking stock

We have been gathering statistical data on our ranches for some time now. Reports based on this information provide us with extremely useful feedback on what we have achieved and how we are improving (see the table).

Note that we are running close to 1 million jobs a month, which collectively consume in the range of 800,000 CPU hours. At the same time, we have managed to keep the average queuing time for short jobs to around three minutes. In fact, in the past year, approximately 50 percent of all jobs that ran in the ranch domain waited in the queue for less than one minute.

In addition, the average month-long utilization of general compute servers has maintained greater than 95 percent utilization during the past year. And in fact, for over a quarter of that time period, the utilization has been essentially 100 percent. In comparison, a user desktop machine in a typical design group has an average utilization metric of somewhere between 15 percent and 20 percent. We achieve this high utilization with a steady 24-hour-a-day stream of medium-to-large batch jobs, which we continuously fine-tune based on the measurements and key reporting mechanisms in our resource-sharing software.

This remarkable uptime also results from the highly automated system administration and preventative maintenance procedures we have put in place. Taking a standardized approach to all the compute resources on the ranch allows for quick and easy distribution of new patches and upgrades when required.

Back in the saddle

Of course, we can't rest on our laurels. We are always looking for ways to improve and expand our ranch capabilities. Consequently, the configuration of the ranch constantly fluctuates and evolves as we add new hardware and software and phase out older equipment and applications.

Looking to the immediate future, our biggest challenge will include the upgrade of all of our EDA applications to make them compliant with our current 64-bit operating system, Solaris 7, and the 64-bit machines that run it. Unfortunately, very few EDA vendors today are producing Solaris 7-certified tools. To ensure compatibility, therefore, we are beta-testing Solaris 7 on critical EDA tools. We are also actively working with our vendors to help them deliver native 64-bit code in as short a time as possible.

Keeping pace with design complexity provides a most compelling reason to focus our efforts on upgrading the tools to 64-bit compatibility. As processors have quickly increased in size, the traditional methods for design begin to break down. At one time, designers could partition processor designs into megacells--each one carefully designed and exhaustively verified--then stitch them together to create a functional design. However, the effects of deep-submicron parasitics are forcing us to examine the overall system performance. The amount of design data now required quickly overwhelms any 32-bit EDA tools and requires breaking the verification task into sub-groups. This situation also limits how much of the layout the tools can examine at any one time.

We continue to improve the ease of use and speed of access to the compute ranch. Toward this end, we are adding a layer of network switches between the ranch and the users' desktops, which will improve performance and keep all traffic management within the ranch. We will also be expanding the compute resources themselves; some resources will reside on the East Coast with our Boston design group, and the remainder will reside in California. Our goal challenges us to have as many as 10,000 CPUs in our computing environment in the foreseeable future.

The exacting demands of today's electronics industry are requiring companies to revise their methods for designing next-generation processors. By exploiting the strengths of network-centric compute ranches, Sun Microsystems continues to move toward an electronic design practice equipped to respond to the needs of the increasingly complex EDA marketplace.

Dwayne Lee is the engineering manager of the Intranet Distributed Engineering Group at Sun Microsystems, Inc. (Mountain View, Calif.). Prior to joining Sun 3 years ago, he was with National Semiconductor for 8 years and Sony Microelectronics for 2 years.

To voice an opinion on this or any other article in Integrated System Design, please e-mail your comments to jeff@isdmag.com.