Point-tracking model 170× slower on T4 than A100—bottleneck mystery

A developer encountered a severe performance disparity when deploying a point-tracking model across two NVIDIA GPU generations. On an A100, the model processes a half-video in approximately 0.5 seconds; on a T4, the same operation takes approximately 85 seconds—a ~170× slowdown. The model operates on video clips of 47 frames at 256×256 pixel resolution, processed one at a time (batch 1), and runs entirely in FP32 precision.

The architecture combines two key components: it constructs local 4D correlation volumes to perform dense frame-to-frame matching, then feeds the results through transformer layers to capture temporal relationships. This combination of memory-intensive dense matching and sequence-modeling computation creates a workload that may expose differences between the GPU generations.

The developer has systematically ruled out many common causes. GPU utilization during inference maxes out at 99% according to nvidia-smi, confirming the hardware is fully engaged rather than idling. The model is confirmed to reside on the GPU (torch.cuda.is_available() returns True and device prints "cuda"). Enabling torch.backends.cudnn.benchmark, a common optimization flag for cuDNN kernel selection, made no difference. Crucially, the slowdown reproduces identically on two separate T4 machines, eliminating isolated driver bugs, firmware issues, or single-machine misconfiguration as explanations.

The developer sought community input on what might cause such an extreme bottleneck, noting that while a meaningful gap between card generations is expected, 170× appears too large to account for through generational hardware differences alone. The puzzle points toward either T4-specific architectural limitations (such as memory bandwidth constraints when building large 4D volumes or less efficient tensor core implementations), inefficient cuDNN or CUDA kernels for this workload on older hardware, or operations within the model that scale particularly poorly on T4's feature set.

The 170× slowdown between T4 and A100 is unusually large and points to a structural mismatch rather than linear performance degradation. Both GPUs show maxed utilization, so the bottleneck is not compute idleness; instead, it likely reflects architectural differences in how the two hardware generations handle the model's specific operations. T4s rely on different tensor core designs and memory subsystems compared to A100, and dense 4D correlation volumes combined with transformer inference may stress one or both of these in ways that do not scale linearly with raw FLOP differences. The fact that two independent T4 machines exhibit identical slowdown rules out isolated driver or firmware problems and suggests the issue is inherent to the T4 GPU architecture or the cuDNN/CUDA kernel implementations for this workload on older hardware.

No comments yet. Be the first to share your thoughts!