This is short post to explain the impact that GPU positioning can have on training time and suggest how to position your GPUs for better performance when you mix blower-style and non-blower GPUs. Consider a deep learning workstation with the following four GPUs:
- 1 EVGA (non-blower-style, Black Edition) RTX 2080 ti (~$1160)
- 2 GIGABYTE (blower-style) RTX 2080 ti (~$1120 each)
- 1 NVIDIA TITAN RTX (~$2500)
We’ll look at a few ways to order these GPUs inside your deep learning machine and evaluate which is best. For a benchmarking comparison of these GPUs on ImageNet and CIFAR [read this].
Consider two ways to position the four GPUs listed above.
|Position 1: TITAN RTX on bottom (GOOD)||Position 2: TITAN RTX on top (BAD)|
The TITAN RTX is the fastest GPU in the system and consumes the most power (280W versus 250W for 2080 ti) and gives off the most heat. The positioning in the figure on the left is good because the TITAN RTX fans are on the bottom and positioning the TITAN RTX on the bottom allows heat to dissipate easily and reduce thermal throttling. However, in the figure on the right, the fans on the bottom of the TITAN RTX are block by the blower-GPU below. This causes the hottest GPU in the system to literally cook itself as the heat has no where to go.
Benchmark comparison on ImageNet
To see this in practice, I benchmarked training ResNet-152 (batch size 32) on ImageNet for both of these GPU layouts. Reported is the average milliseconds/image for the first epoch of training, with all four GPUs running simultaneously. If you’d like to reproduce or learn more about how this benchmark is computed, [read this].
|GPU||Position 1:||ms/image||Position 2:||ms/image|
|EVGA (non-blower) RTX 2080 ti||top||9.66||bottom||8.84|
|GIGABYTE (blower) RTX 2080 ti||mid-top||9.22||mid-bottom||13.59|
|GIGABYTE (blower) RTX 2080 ti||mid-bottom||10.78||mid-top||9.47|
|NVIDIA TITAN RTX||bottom||8.22||top||21.69|
As you can see, positioning the TITAN RTX on top more than doubles (8.22 ms —> 21.69 ms) training time due to blocking the fans with the GPUs below. To my surprise, the manual that comes with the TITAN RTX recommends installing it in the top slot on your motherboard, presumably because NVIDIA assumes you have no other GPUs in your system.
General principles for GPU positioning
In general, we have the following:
|Non-blower on bottom (GOOD)||Non-blower on top (BAD)|
|All blower-style GPUs (GOOD)||Mixed (OK), but the top non-blower is (BAD)|
Blower-style GPUs should always be placed in the middle as they blow air out of the side, not the top or bottom.
If you mix blower-style and non-blower GPUs, install the non-blower GPU in the bottom slot on your motherboard.