GeForce GTX Titan Two-Way SLI Scaling: PCIe 2 vs. PCIe 3

Back when I had Origin’s tri-Titan equipped Genesis in house, I received a few requests for benchmarks with PCIe 3 enabled. Because we tested the system with its default out of the box settings, the system was configured by default to only use PCIe 2 since NVIDIA does not officially support PCIe 3 on SNB-E based systems. As a result there was some interest in whether PCIe 3 would improve performance at all compared to PCIe 2, due to the high amount of PCIe bus traffic generated by three GTX Titan cards working together. IVB/Haswell is of course limited to 16 PCIe 3 lanes, whereas if PCIe 3 is enabled SNB-E systems have more than twice that with a combined total of 40 lanes, giving SNB-E a large bandwidth advantage.

As it turned out, the Intel DX79SR used in the Genesis supported PCIe 3, but only on the 2 full-fledged x16 slots. The third x16 slot (electrically x8) won’t operate beyond PCIe 2 speeds even after forcing it in NVIDIA’s drivers, which is unfortunately a bit of a double whammy since it’s the slot that’s already lacking in bandwidth. That dashed our hopes of a PCIe 3 vs. PCIe 2 comparison for tri-SLI Titans, and as a result the subject never reached print.

However in the last couple of weeks I’ve received several requests for two-way SLI Titan/780 PCIe performance benchmarks, which we did happen to collect while looking into tri-SLI and merely never published. So with that in mind we’re going to finally publish that data, since there’s clearly more interest in the matter than I initially expected.

As a heads up this data is a bit stale; it was taken back in April with NVIDIA’s 314.22 drivers. So the primary goal here is to compare two-way SLI Titan performance on PCIe 2 versus PCIe 3.

At 2560x1440 there’s very little change. The only game to see any kind of notable improvement is Total War: Shogun 2, which picks up 5%, though this is going from 115fps to 120fps. Otherwise PCIe 3 does not confer a distinct performance advantage here.

At 5760 things become more interesting, although again the performance gains aren’t particularly huge. Every game benefits at least marginally from PCIe 3 being enabled; Total War: Shogun 2 once again shows the greatest benefit at 7%, while Battlefield 3 picks up 5%, and Crysis 3 a hair over 2%.

Admittedly this is a small sample set, but it does paint a distinct picture of the PCIe bus being a bottleneck when using surround modes. NVIDIA has never gone into great detail on how their surround mode works, but based on this and previous data we’ve long assumed that they’re shuffling a significant amount of data around via the PCIe bus in order to keep the cards in sync and to give each card the portion of the frame buffer it will be responsible for sending to its attached display(s). This is as opposed to 2560x1440 and other single monitor modes, where the master card merely needs to collect completed frames and there’s no further redistribution involved.

In any case, while NVIDIA does not officially support PCIe 3 on SNB-E systems, between the Genesis and our own testbed we’ve yet to run into an issue in using it. Consequently we can’t see any good reason not to enable it, especially in surround mode setups. Based on our data the performance gains aren’t going to be huge (at least not in two-way SLI), but it’s free performance being left on the table. At the very least it’s worth a shot, since NVIDIA makes it easy to turn PCIe 3 support on and off.