The over-provisioning of NAND flash memory in solid state drives (SSDs) and flash memory-based accelerator cards (cache) is a required practice in the storage industry owing to the need for a controller to manage the NAND flash memory. This is true for all segments of the computer industry—from ultrabooks and tablets to enterprise and cloud servers.
Essentially, over-provisioning allocates a portion of the total flash memory available to the flash storage processor, which it needs to perform various memory management functions. This leaves less usable capacity, of course, but results in superior performance and endurance. More sophisticated applications require more over-provisioning, but the benefits inevitably outweigh the reduction in usable capacity.
The Need for Over-provisioning NAND Flash Memory
NAND flash memory is unlike both random access memory and magnetic media, including hard disk drives, in one fundamental way: there is no ability to overwrite existing content. Instead, entire blocks of flash memory must first be erased before any new pages can be written.
With a hard disk drive (HDD), for example, that act of “deleting” files affects only the metadata in the directory. No data is actually deleted on the drive; the sectors used previously are merely made available as “free space” for storing new data. This is the reason “deleted” files can be recovered (or “undeleted”) from HDDs, and why it is necessary to actually erase sensitive data to fully secure a drive.
With NAND flash memory, by contrast, free space can only be created by actually deleting or erasing the data that previously occupied any block of memory. The process of reclaiming blocks of flash memory that no longer contains valid data is called “garbage collection.” Only when the blocks, and the pages they contain, have been cleared in this fashion are they then able to store new data during a write operation.
The flash storage processor (FSP) is responsible for managing the pages and blocks of memory, and also provides the interface with the operating system’s file subsystem. This need to manage individual cells, pages and blocks of flash memory requires some overhead, and that in turn, means that the full amount of memory is not available to the user. To provide a specified amount of user capacity it is therefore necessary to over-provision the amount of flash memory, and as will be shown later, the more over-provisioning the better.
The portion of total NAND flash memory capacity held in reserve (unavailable to the user) for use by the FSP is used for garbage collection (the major use); FSP firmware (a small percentage); spare blocks (another small percentage); and optionally, enhanced data protection beyond the basic error correction (space requirement varies).
Even though there is a loss in user capacity with over-provisioning, the user does receive two important benefits: better performance and greater endurance. The former is one of the reasons for using flash memory, including in solid state drives (SSDs), while the latter addresses an inherent limitation in flash memory.
The equation for calculating the percentage of over-provisioning is rather straightforward:
For example, in a configuration consisting of 128 Gigabytes (GB) of flash memory total, 120 GB of which is available to the user, the system is over-provisioned by 6.7 percent, which is typically rounded up to 7 percent: