Nvidia developed a cluster-gathering scheme called IndeX. But they didn't stop there. Nvidia now lets you gang up multiple GPUs within a cluster. This GPU-to-GPU inter-linking connection is called NVlinks.
Pascal (the subject of a separate discussion/article) has many interesting features, not the least of which is build-in, or rather I should say, built-on, memory. Pascal will have memory stacked on top of the GPU. That not only makes a tidier package, more importantly it will give the GPU 4x higher bandwidth (~1 TB/s), 3x larger capacity, and 4x more energy efficient per bit.
Basically the already high-speed GPU to video memory bandwidth will go up four orders of magnitude. That alone will help speed up things, but Nvidia took it one-step further and added GPU-to-GPU links that allow multiple GPUs to look like one giant GPU.
Nvidia's NVLinks connecting four GPUs and the CPU (Nvidia)
Today a typical system has one or more GPUs connected to a CPU using PCI Express. Even at the fastest PCIe 3.0 speeds (8 Giga-transfers per second per lane) and with the widest supported links (16 lanes) the bandwidth provided over this link pales in comparison to the bandwidth available between the GPU and its system memory.
NVLink addresses this problem by providing a more energy-efficient; high-bandwidth path between the GPU and the CPU at data rates 5 to 12 times that of the current PCIe Gen3. NVLink will provide between 80 GB/s and 200 GB/s of bandwidth.
The numbers are astronomical, and they need to be because the data sizes and rates aren't slowing down and are also astronomical. And, just to make a pun, this now improves astrophysics and astronomy research too. (Nvidia's GPU-compute systems are being used to tease out the beginning of the big bang -- now that's truly BIG data).
And the really good news? The costs and power requirements are not astronomical, in fact, the power requirements are less than a tenth of what they would have been (for an equivalent amount of compute resource) four years ago.
This is the opening phase of a new threshold in understanding of enormously complex systems like weather, geophysics, mechanics, and the human body.
Ten years from now our lives will be so much better because of the wonders in medical science and the management of multifaceted systems, we'll look back on 2014 with sympathy and say, how did they ever get along with such primitive tools?