A simpler design is needed to minimize the amount of hardware and networking cluttering the server rack. Managing the rack as a separate unit with shareable resources will deliver the most efficient infrastructure. PCIe has numerous intrinsic benefits. It is low power, widely accepted, economical, and integrated at the base system level. The CPU and chipset already have integrated PCIe, and applications support the protocol natively. The extension of PCIe out of the server chassis is a natural progression of the existing design.
The first steps of widespread adoption of PLX Technology's ExpressFabric will involve a reduction in the amount of networking equipment. Current designs leverage multiple network fabrics to achieve a very similar task. This leads to excess hardware, cost, power consumption, cabling, and complexity.
Removing this extra layer of hardware greatly simplifies design. Utilizing PCIe as a converged fabric provides one high-performance low-latency link between all components. 32Gb/s links (x4 in PCIe parlance) are scalable to 64G (x8) and 128G (x16), providing a robust interconnect that can adjust to different requirements. Copper and optical QSFP+ interconnects are standard fare in the datacenter and offer a proven and reliable means of connection.
ExpressFabric forgoes use of expensive NICs, LOMs, and HBAs, instead employing a Retimer. Retimers boost PCIe out of the chassis, consume less than one watt of power, and cost roughly $5. These connections are routed to the ExpressFabric top-of-rack switch. This switch houses an HBA and NIC for communication outside of the rack. In this instance, simple Retimers have replaced 16 HBAs and 16 NICs. This leads to radically lower power consumption, component cost, and complexity.
Perhaps most importantly, the ExpressFabric top-of-rack switch houses a NIC and an HBA. The servers react to these resources as they would if they were physically installed in the server. This allows an optimum use of networking resources. This fully utilizes the hardware, instead of the hardware simply standing idle or only operating at a portion of its capability. Ethernet will provide the backbone and Infiniband will be used in HPC applications.
Measured on a Gbps basis, the ExpressFabric converged networking solution provides a radical reduction in cost and power consumption over both traditional Ethernet and Infiniband solutions. Peripheral advantages, such as lower heat output from the servers, simplified management, and reduced maintenance, will add to the benefits of these deployments.