New QUEST attention mechanism stabilizes Transformer training by constraining keys to hyperspheres while maintaining flexible attention control

QUEST (QUEry-modulated Spherical aTtention) addresses training instability caused by unbounded query and key vector norms in standard Transformer attention

The method uses softmax on scaled dot products but constrains keys to a hyperspherical latent space while allowing queries to modulate attention sharpness

QUEST serves as a drop-in replacement for standard attention in existing Transformer models without architectural changes

Researchers demonstrated the problem occurs even in simple Transformers when spurious patterns are easily learnable from data

The approach is validated primarily on vision applications while showing generality across other domains