Optimizing PCIe SSD performance

The importance of scalability
A second key aspect of creating large, fast storage systems is scalability. This is enabled by the flexibility, expandability, and bandwidth of the interconnect. Systems have differing needs—some offer a very large array of memory devices needing many connection ports, while others are better served by aggregating relatively fewer memory subsystems, with each one providing vastly more bandwidth.

This variability matches well with what PCI Express can offer, since the standard allows each connection port to bundle a flexible number of high-speed lanes. The basic bandwidth of PCI Express Gen 3 is 8 Gbps per lane per direction, and you can combine these individual lanes to form reasonable ports that offer, for example, 64Gbps of bidirectional bandwidth with a x4 link (4 lanes aggregated into a single port; see figure 3).



Now that we understand the impact that low latency has on general-purpose system performance, there is a class of application in which bandwidth is the primary performance bottleneck. In applications like database loads and backup/archival, large datasets are streamed without interruption once the transfer has started. PCIe can accommodate very high transfer rates by aggregating lanes into an appropriate number of ports; for example, a x16 Gen-3 port can handle up to 128 Gbps in a single direction (for streaming), and twice that if the tasks can be done in parallel. This combination of scalable bandwidth and port flexibility is perfectly matched to the inherent strengths of the SSD subsystems.

Over the past five years, the prevailing transfer mechanism for SSDs has been through a host bus adapter (HBA). Moving forward, SSD form factors are expected to radically change, based on emerging trends at both the component and system levels. New SSD form factors in the shape of 2.5-in. Drives are emerging to enable incremental scalability in terms of performance and capacity, and improved supportability (e.g. hot swap). Next-generation servers are enabling this through physical changes, such as SFF-8639 connectors, as well as industry-standard protocols like NVM Express and SCSI Express. The result will drive a need for host PCIe lanes to be aggregated to many sets of PCIe Gen 3 at x4 widths, and may be further bifurcated to two sets of x2 widths for failover.

Additionally, the emerging SATA Express specification may also play a role in the enterprise. While initially targeted toward client environments to take advantage of the PCIe interface, the popularity of SATA Express form factors (mast, NGFF, used) will potentially result in enterprise server adoption—for example, a PCIe Gen3 x2 configuration SSD module or embedded device.