SAN JOSE, Calif. – PLX Technology is developing a variant of its PCI Express switches to act as a fabric for a variety of servers and network appliances. The effort is one of several proposals to expand the reach and role of PCIe, now the subject of a lively debate.
Server designers at Facebook want to use a PCIe fabric as the basis for future processor cards that ease the task of upgrading CPUs in the thousands of racks in its large data centers. Currently the company uses server cards that include relatively expensive 10 Gbit/second Ethernet links.
“Our goal is to use only PCIe in the rack so we can refresh every component on its own natural life cycle,” said Amir Michael, a senior hardware designer at Facebook which leads the Open Compute Project, trying to set standards in server design.
“I believe [a PCIe fabric] is one of the next big frontiers for servers,” Michael said. “It has a lot of benefits, but also some technology challenges because PCIe was not designed for that and there are no long distance, cost-effective PCIe components now,” he added.
PLX (San Jose) hopes to fill the gap with a chip and software it currently has in development with unnamed partners. It includes “standards-compliant extensions to its switches” and is aimed at data-center and high-performance computing servers as well as network appliances such as dedicated storage systems, said a PLX representative.
The PLX approach will offer twice the throughput of 10G Ethernet with slightly lower CPU utilization. PLX is working with standards groups to get its approach adopted.
Sources at two server makers said the PLX effort is one of several similar proposals in the industry. However, none are actively under consideration at the PCI Special Interest Group, said one senior I/O engineer who asked not to be named.
“Perhaps for some purpose-built applications, [PLX] will find an interesting market, but for the volume space of cloud and enterprise data centers, the world is dominated by Ethernet,” the engineer said. “With the rise of processors with integrated Ethernet NICs across multiple design points, the technical challenges may be the least to overcome,” he said.
“Convincing customers to implement two fabrics, two management domains, and etc., will be a challenge,” the engineer said.
He noted Infiniband has faced similar challenges and thus “still remains a niche for purpose-built environments.” In addition, Intel scrapped several years ago its Advanced Switching initiative that tried to create a version of PCIe acting as a broad clustering and network fabric, he added.
Intel recently said it is developing a clustering interconnect it will put on its future server processors, borrowing techniques from many technologies including Ethernet, it said. Separately, archrival Advanced Micro Devices is promoting the Freedom Fabric acquired with startup SeaMicro.
David Patterson, known for his pioneering research that led to RAID, clusters and more, is part of a team at UC Berkeley that recently made its RISC-V processor architecture an open source hardware offering. We talk with Patterson and one of his colleagues behind the effort about the opportunities they see, what new kinds of designs they hope to enable and what it means for today’s commercial processor giants such as Intel, ARM and Imagination Technologies.