This is short post to explain the impact that GPU positioning can have on training time and suggest how to position your GPUs for better performance when you mix blower-style and non-blower GPUs. Consider a deep learning workstation with the following four GPUs:

We’ll look at a few ways to order these GPUs inside your deep learning machine and evaluate which is best. For a benchmarking comparison of these GPUs on ImageNet and CIFAR [read this].

GPU Positioning

Consider two ways to position the four GPUs listed above.

Position 1: TITAN RTX on bottom (GOOD) Position 2: TITAN RTX on top (BAD)

The TITAN RTX is the fastest GPU in the system and consumes the most power (280W versus 250W for 2080 ti) and gives off the most heat. The positioning in the figure on the left is good because the TITAN RTX fans are on the bottom and positioning the TITAN RTX on the bottom allows heat to dissipate easily and reduce thermal throttling. However, in the figure on the right, the fans on the bottom of the TITAN RTX are block by the blower-GPU below. This causes the hottest GPU in the system to literally cook itself as the heat has no where to go.

Benchmark comparison on ImageNet

To see this in practice, I benchmarked training ResNet-152 (batch size 32) on ImageNet for both of these GPU layouts. Reported is the average milliseconds/image for the first epoch of training, with all four GPUs running simultaneously. If you’d like to reproduce or learn more about how this benchmark is computed, [read this].

GPU Position 1: ms/image Position 2: ms/image
EVGA (non-blower) RTX 2080 ti top 9.66 bottom 8.84
GIGABYTE (blower) RTX 2080 ti mid-top 9.22 mid-bottom 13.59
GIGABYTE (blower) RTX 2080 ti mid-bottom 10.78 mid-top 9.47
NVIDIA TITAN RTX bottom 8.22 top 21.69

As you can see, positioning the TITAN RTX on top more than doubles (8.22 ms —> 21.69 ms) training time due to blocking the fans with the GPUs below. To my surprise, the manual that comes with the TITAN RTX recommends installing it in the top slot on your motherboard, presumably because NVIDIA assumes you have no other GPUs in your system.

General principles for GPU positioning

In general, we have the following:

Non-blower on bottom (GOOD) Non-blower on top (BAD)
All blower-style GPUs (GOOD) Mixed (OK), but the top non-blower is (BAD)

Blower-style GPUs should always be placed in the middle as they blow air out of the side, not the top or bottom.

Golden rule

If you mix blower-style and non-blower GPUs, install the non-blower GPU in the bottom slot on your motherboard.