Researchers discover that normalization-free transformers achieve better gradient stability through subcritical signal propagation, offering insights into deep neural network initialization.

Study analyzes signal propagation in transformers using averaged partial Jacobian norm (APJN) to measure gradient amplification across layers

Theory extends to bidirectional attention and permutation-symmetric token configurations by deriving recurrence relations for activation statistics

Pre-LayerNorm architectures show power-law APJN growth, while tanh-like nonlinearities exhibit stretched-exponential growth indicating subcritical behavior

Findings apply to Dynamic Tanh (DyT) and Dynamic erf (Derf) transformers, explaining why these normalization-free designs achieve better training stability

Predictions match empirical measurements in deep vision transformers, bridging theoretical understanding and practical deep network behavior