PCI-SIG Forms Optical Workgroup - Lighting The Way To PCIe's Future
by Ryan Smith on August 2, 2023 11:00 AM EST- Posted in
- CPUs
- PCIe
- PCI-SIG
- Optical
- Silicon Photonics
The PCI-Express interconnect standard may be going through some major changes in the coming years, based on a new announcement from the group responsible for the standard. The PCI-SIG is announcing this morning the formation of a PCIe Optical Workgroup, whose remit will be to work on enabling PCIe over optical interfaces. And while the group is still in its earliest of stages, the ramifications for the traditionally copper-bound standard could prove significant, as optical technology would bypass some increasingly stubborn limitations of copper signaling that traditional PCIe is soon approaching.
First released in the year 2000, PCI-Express was initially developed around the use of high-density edge connectors, which are still in use to this day. The PCIe Card Electromechanical specification (CEM) defines the PCIe add in card form factors in use for the last two decades, ranging from x1 to x16 connections.
But while the PCIe CEM has seen very little change over the years – in large part to ensure backward and forward compatibility – the signaling standard itself has undergone numerous speed upgrades. Including the latest PCIe 6.0 standard, the speed of a single PCIe lane has increased by 32-fold since 2000 – and the PCI-SIG will double that once more with PCIe 7.0 in 2025. As a result of increasing the amount of data transferred per pin by such a significant amount, the literal frequency band width used by the standard has increased by a similar degree, with PCIe 7.0 set to operate at nearly 32GHz.
In developing newer PCIe standards, the PCI-SIG has worked to minimize these issues, such as by employing alternative means of signaling that don’t require higher frequencies (e.g. PCIe 6 with PAM-4), and the use of mid-route retimers along with materials improvements have helped to keep up with the higher frequencies the standard does use. But the frequency limitations of copper traces within a PCB have never been eliminated entirely, which is why in more recent years the PCI-SIG has developed an official standard for PCIe over copper cabling.
Still in the works for late this year, the PCIe 5.0/6.0 cabling standard offers the option of using copper cables to carry PCIe both within a system (internal) and between systems (external). In particular, the relatively thick copper cables have less signal loss than PCB traces, overcoming the immediate drawback of high frequency comms, which is the low channel reach (i.e. short signal propagation distance). And while the cabling standard is designed to be an alternative to the PCIe CEM connector rather than a wholesale replacement, its existence underscores the problem at hand with high frequency signaling over copper, a problem that will only get even more challenging once PCIe 7.0 is made available.
PCIe Insertion Loss Budgets Over The Years (Samtec)
And that brings us to the formation of the PCI-SIG Optical Workgroup. Like the Ethernet community, which tends to be at the forefront of high frequency signaling innovation, PCI-SIG is looking towards optical, light-based communication as part of the future for PCIe. As we’ve already seen with optical networking technology, optical comms offers the potential for longer ranges and higher data rates vis-à-vis the vastly higher frequency of light, as well as a reduction in power consumed versus increasingly power-hungry copper transmission. For these reasons, the PCI-SIG is forming an Optical Workgroup to help develop the standards needed to supply PCIe over optical connections.
Strictly speaking, the creation of a new optical standard isn’t necessary to drive PCIe over optical connections. Several vendors already offer proprietary solutions, with a focus on external connectivity. But the creation of an optical standard aims to do just that – standardize how PCIe over fiber optics would work and behave. As part of the working group announcement, the traditionally consensus-based PCI-SIG is making it clear that they aren’t developing a standard for any single optical technology, but rather they are aiming to make it technology-agnostic, allowing the spec to support a wide range of optical technologies.
But the relatively broad announcement from the PCI-SIG doesn’t just stop with optical cabling as a replacement for current copper cabling, the group is also looking at “potentially developing technology-specific form factors.” While the classic CEM connector is unlikely to go away entirely any time soon – the backwards and forwards compatibility is that important – the CEM connector is the weakest/most difficult way to deliver PCIe today. So if the PCI-SIG is thinking about new form factors, then it’s likely the Optical Workgroup will at least be looking at some kind of optical-based successor to the CEM. And if that were to come to pass, this would easily be the biggest change in the PCIe specification in its 23+ year history.
But, to be sure, if any such change were to happen, it would be years down the line. The new Optical Workgroup has yet to form, let alone set its goals and requirements. With a broad remit to make PCIe more optical-friendly, any impact from the group is several years away – presumably no sooner than making a cabling standard for PCIe 7.0, if not a more direct impact on a PCIe 8.0 specification. But it shows where PCI-SIG leadership sees the future of the PCIe standard going, assuming they can get a consensus from their members. And, while not explicated stated in the PCI-SIG’s press release, any serious use of optical PCIe in this fashion would seem to be predicated on cheap optical transceivers, i.e. silicon photonics.
In any case, it will be interesting to see what eventually comes out of the PCI-SIG’s new Optical Workgroup. As PCIe begins to approach the practical limits of copper, the future of the industry’s standard peripheral interconnect may very well be to go towards the light.
Source: PCI-SIG
16 Comments
View All Comments
konbala - Wednesday, August 2, 2023 - link
Computer audiophiles' dream come truemode_13h - Wednesday, August 2, 2023 - link
Why? You can already transmit 24/96 to an outboard DAC over toslink. If you need something like Dolby Atmos and want optical isolation, there are optical transceivers you can get for HDMI.edzieba - Wednesday, August 2, 2023 - link
Also PIC-SIG: OCuLink? What OCuLink?nfriedly - Wednesday, August 2, 2023 - link
It would be pretty awesome if PIC-SIG made the official cabling standard compatible with OCuLink. But if not, I could see passive adapters as a possibility.speedping - Wednesday, August 2, 2023 - link
Would this enable cloud computing with modular GPUs? (Like NVMe-oF did to SSD storage?)How far away can something like a GPU be until the added latency makes it unusable? Each meter adds 3.3 nanoseconds of latency due to the speed of light
mode_13h - Wednesday, August 2, 2023 - link
Presumably anything they do would apply equally to CXL, which already supports switch fabrics. It's therefore entirely plausible to see rack-scale disaggregated GPU compute setups.I think there are still pragmatic reasons to keep the GPUs in the same rack, however. Otherwise, the amount of bandwidth you'd need between racks would get pretty nuts.
Jorgp2 - Wednesday, August 2, 2023 - link
Those have been a thing for at least a decade.You could get PCIE expansion systems just like you could with storage.
mode_13h - Wednesday, August 2, 2023 - link
I tihnk PCIe over optical shouldn't be just inserting a pair of optical transceivers at each end, but should revisit fundamental aspects of the signalling to properly adapt it for a fiberoptics medium. I sure hope whatever they do can scale at least half as well as PCIe has done so far, and that's unlikely if they make too many short-term oriented compromises.> ranging from x1 to x16 connections.
I thought it went up to x32, even though we don't often see them in practice.
watersb - Wednesday, August 2, 2023 - link
I don’t work with modern server-class hardware, but I have noticed how servers are now cables for internal connections like data link between CPU sockets.Short cables where both ends terminate on the same circuit board — replacing long traces on the circuit board with shorter travels to a high density plug or socket of some kind, then over a cable, to another such connector.
Sure, for a decade that the industry has been using PCIe lanes for the CPU to talk to the motherboard chipset; in theory that traffic could be carried over a short cable or riser adapter, but such an approach was quite unusual.
But now PCIe bus traffic has become so high-frequency that circuit boards cannot deliver the signals.
Maybe this is obvious to everyone here, but it’s still new to me, and has not yet become necessary for consumer PCIe 5…
But how long until your gaming rig needs CXL to talk to RAM?
mode_13h - Thursday, August 3, 2023 - link
> how long until your gaming rig needs CXL to talk to RAM?Once the main memory turns into a stack sitting on the same interposer as the CPU, then CXL will be the natural option for memory expansion.