New Triton kernel reduces attention dispatch overhead by 40% to unlock real-world speedups from Vision Transformer token pruning

Token pruning methods promise quadratic FLOP reductions in Vision Transformers but fail to translate into actual wall-clock speedups due to dispatch overhead consuming 60-90 microseconds

FlashAttention-2's varlen and PyTorch's NestedTensor SDPA struggle with short post-pruning sequences (≤197 tokens) where actual computation finishes in single-digit microseconds

Researchers developed a lightweight bidirectional Triton attention kernel with a 40-microsecond dispatch floor—roughly 1.5x lower than FlashAttention-2 varlen

Complete pack-attend-unpack pipeline achieves up to 2.24x end-to-end throughput improvement over padded PyTorch SDPA baselines