cover story

Putting a RAID Controller into a System-level FPGA A high level of cooperation between software and hardware design teams facilitated the speedy creation of an FPGA-based video RAID controller. by Michael H. Anderson

The term "system on a chip" often brings to mind an ASIC with millions of gates, combining processors, memory, RAM, ROM, and digital logic and analog circuits. However, high gate counts and a mix of technologies isn't really necessary. By combining the talents of hardware and software engineers, Medea Corp. recently achieved a system-on-a-chip integration using a standard FPGA.

The system that we integrated into a single chip was a redundant array of independent disks (RAID) controller. A RAID o controller combines the data from multiple disk drives, presenting the appearance of a single, fast disk to a host computer.

Through cooperation between the hardware and software engineers and the strong similarities between C and Verilog, we reached the system-on-a-chip level of integration. The FPGA itself also contributed to the project's success by helping us to pinpoint errors quickly. We created entire subsystems that reflected coordinated efforts on the part of the hardware and software teams to duplicate and eliminate design flaws. Instead of the standard "over the fence" attitude between the software and hardware teams, each team utilized the other team's skill set to make short work of some subtle problems.

Background

A RAID system typically connects multiple drives--commonly four--through a combination of host, drive, and expansion interfaces (see Figure 1). A host computer connects to the RAID system through a cable attached to the host interface. Data is striped--distributed serially--across the drives, allowing them to aggregate their data rate in response to a host request. For expandability, an expansion interface allows a second RAID system to attach to the first. The secondary port serves just for expansion or can allow simultaneous access to the RAID from more than one host computer. RAID systems are great for editing digital video, since the high data rate allows for very little compression, resulting in very high quality video.

The controller architecture for that type of RAID system typically uses control logic and FIFOs to distribute data to and from the host and drives (see Figure 2). A host interface chip contains the protocol necessary to connect the RAID controller to a host. Overseen by a memory controller, the data travels from the host interface chip into a buffer memory, then moves from the buffer memory to the disk interface chips or to the expansion interface chip. To keep performance high, a dedicated data bus moves the data. A local microprocessor uses a control bus to coordinate the actions of the host interface chip, the disk interface chips, and the expansion interface; it also handles initialization and error recovery.

RAIDing the programmables

We set out to build a RAID controller for video and graphical editing applications that would achieve high performance and low cost. We considered a traditional design using standard protocol chips and a microprocessor, but the resulting costs were too high. As an alternative, we decided to use an FPGA to try to reduce component count and overall cost. Our first targets for cost reduction were the numerous drive protocol chips.

Figure 1	The RAID system
In a typical dual-ported RAID system configuration, a PC connects directly to the first RAID's host interface. Each subsequent RAID is connected to the expansion interface of the previously connected RAID. Each RAID manages four or more separate disk drives.

We used Verilog to design the protocol logic for the four drive protocol chips. We chose an Altera EPF6016, the lowest-cost FPGA we could find that could hold the logic. Using the 208-pin package, we could directly connect the four drive buses to a single chip, replacing four of the disk interface chips with a single FPGA.

What happened during that step became the key to achieving a full system-on-a-chip integration. Our hardware designers were less familiar with the drive protocol than were our software designers. Eventually, after a great deal of communication, our software engineers, fluent in C, were exposed to the Verilog code produced by our hardware engineers. To their surprise, they could understand the majority of the hardware description.

C and Verilog share a great deal of syntax--the conditional assignment statement, for example. In C, you can assign a variable based on the "truth" of a Boolean equation. For example, the statement a=b?c:d assigns the variable "a" the value "c" if "b" is true or the value "d" if "b" is false. Verilog uses the same syntax to describe a multiplexer. Other strong similarities include the if, else, and switch/case statements. After a little schooling in implied registers, clocks, and state machine descriptions, our software engineers made a real contribution to the development of Verilog code.

Handling state machines would normally have been the tricky part of the drive protocol. Our software engineers, however, well versed in the use of procedural languages, made short work of the state machines required for the drive protocol. Once they grasped the basics of building a sequential state machine in Verilog, they were eager to discover how much of the rest of their software could be implemented in Verilog instead of C.

The next step of the integration targeted the host interface and expansion interface chips, employing the same protocol in both chips. Using the same techniques we used for the disk interface protocol logic, the hardware and software engineers collaborated to produce a suitable Verilog description that could replace the two chips. Though the development proceeded quickly, the resulting architecture produced a new problem.

Even though we now possessed the logic for the host, the expansion, and the drive protocol, no FPGA that we could afford could integrate the resulting system. To incorporate all the logic, a single chip would need four drive buses, one host bus, one expansion bus, a processor bus, and a memory address and data bus. The required pin count made a single chip too expensive. Using two chips could have worked, but we found a better solution.

Fortunately, the 10K family of chips included an embedded memory called an embedded array block (EAB). For our application, we needed just a couple of 512-byte sectors to serve as a speed-matching FIFO. The EPF10K20 part had six 8-bit-by-256-word memories, from which we created two 16-bit-by-256-word FIFO buffers. Thus we eliminated the need for the external memory address and data bus and were therefore able to move to a more affordable part, a 240-pin EPF10K20.

Altera supplied a megafunction that completely replaced the external memory controller and buffer. We specified the FIFO width and depth, then attached the FIFO control signals; the megafunction did all the rest. Now our system consisted of a microprocessor subsystem and a single FPGA.

Down to one

Our software engineers, however, weren't done. They came up with a state machine that--with no external software--could automate the flow of commands from the host protocol to the drives. Using only a few registers and fewer than 12 separate states, the machine could execute an entire command, while fitting easily into the EPF10K20. The design would allow us to remove the microprocessor and its ROM and RAM from the board altogether (see Figure 3)--a very attractive cost savings.

Figure 2	The RAID architecture
The RAID controller needs to direct data to and from the host and the multiple drives. The FPGA implementation replaced a number of external interfaces and datapaths.

Close inspection, however, revealed another problem. The state machine could handle normal operations, but not error events. The number of possible states during error recovery proved too great to design a simple machine that would run in the FPGA. To solve that problem, we decided to supply a software utility in the form of a device driver that would run on the host computer. The software could handle the complexities of error recovery, yet not burden the cost of our product with extra hardware.

Removing the microprocessor ROM also removed the source of the code for the FPGA, which is an SRAM-based device. We could have supplied a dedicated ROM for the FPGA code. However, one of our hardware engineers pointed out that if we could program the FPGA over the host bus, we would save the ROM required to program the FPGA at power-up. The FPGA code became part of the device driver that ran in the host. The technique also offered the advantage of allowing us to update the "hardware" (the code in the FPGA) by changing out the software driver. If we found a hardware bug, a customer could download a fix over the Internet instead of having to return the unit to us for rework. Similarly, we decided to add a 16244 to allow the cable from either the host interface or the expansion interface to program the FPGA. The addition increased the flexibility of the system at very little additional cost.

As a result of our design efforts, we reduced our RAID controller from 11 components down to 1.

Tools and dice

We used the Altera Max II+ design tools to simulate the Verilog code. Altera integrates several tools into one environment, including a text editor for creating the source code, a compiler and synthesizer to produce the netlist, pull-down menus to set various logic and synthesis options, a place-and-route tool, a timing analyzer, and a waveform editor to create test vectors. The integrated environment allowed us to oscillate easily between the test vectors and source code, and a built-in hierarchy display let us navigate through the multiple source files by using the mouse.

We used the automatic place-and-route feature with a timing constraint of 33 MHz, making the design work in a medium-speed, "-4" part--the lowest-cost EPF10K20 available. We built a series of test benches using the waveform editor supplied with the tools, and we covered several read/write test cases. When we ran into problems, we simulated individual components of the design separately until we uncovered the source of the problem. The tools made the hierarchy traversal easy, keeping compilation times all under 15 minutes. We also used the floorplan editor to obtain an optimal pin layout. By pinning the FPGA to match the host and drive pin-outs, we limited the board to four layers, fit all the components on a single side, and kept the trace lengths very short. The resulting board was low-cost, low-noise, and very easy to manufacture.

Figure 3	The RAID controller
A traditional RAID electronic architecture uses 11 or more separate electronic components. Two buses interconnect the components: The control bus allows the microprocessor to set the state of other components, whereas the data bus provides a high-speed datapath between the disk interfaces and the host and expansion interfaces.

Our software team wrote the drivers for the product in C++ using the Microsoft-supplied Device Driver Development Kit (DDK). We produced a Mini-Port Driver (MPD) for Windows 95 and 98 and a SYS driver for Windows NT with Microsoft Visual C++ version 5.0. To debug the drivers, we used Numega's Soft Ice, which provides source-level debugging for kernel-mode programs like our disk device drivers. The drivers contain all the code required to initialize the FPGAs and handle all the error recovery for the RAID system. The combined driver code and FPGA code compiled into about 100 kbytes and was small enough to distribute on a floppy disk.

RAID debugging

The implementation and debugging cycle moved extremely well. Since the hardware and software engineers had codesigned the system, tracing the source of problems went very quickly.

We caught the first mistakes when we received our first boards. The schematic mislabeled two signals, misrouting them from the FPGA to each drive connector. However, instead of fixing the problem with blue wires, we went back to the floorplan editor and changed the affected pin assignments. In less than 20 minutes, we were back in business.

Our design contained a few errors that we didn't catch in simulation. To find them, we reprogrammed the FPGA to expose buried logic on pins that connected to a drive connector, then hooked up a logic analyzer in place of the attached disk drive. The technique proved very effective in finding the source of failures. The simultaneous view of the host signals, drive signals, and internal signals made it clear where we made our errors. The ability to quickly recompile and reprogram the FPGA was extremely valuable in that stage of debugging.

The teamwork between the software and hardware engineers was key in detecting the last few subtle, low-frequency errors, which would manifest themselves as data corruption in one form or another. Since the software engineers had a good idea of the hardware architecture, they were able to imagine scenarios that could account for the failure, then write software to stress the conditions. Together with the hardware engineers, they built state machines that could anticipate expected data patterns, then fire a signal when a data corruption was detected. The technique allowed our logic analyzer to get very close to the error event. The common design view and similar language base between the two teams made that kind of cooperation possible.

The resulting product, VideoRAID, went from concept to production in just over five months and cost 75 percent less to manufacture than other products of its type. The FPGA was a major factor in the extremely short development cycle; the high level of integration accounts for the cost difference.

Our real achievement wasn't producing a system-on-a-chip design that required millions of gates to duplicate an existing electronic architecture. Instead, a common language and mental model in our senior hardware and software designers produced a new design approach that realized a system-on-a-chip implementation with a standard FPGA.

Michael H. Anderson is the vice president of engineering and a cofounder of Medea Corp. in Westlake Village, Calif. He was previously senior director of engineering at Micropolis. He holds three patents in storage technology and teaches C, C++, and Verilog at a local community college.

To voice an opinion on this or any Integrated System Design article, please email your message to miker@isdmag.com.

integrated system design  February 1999