PyTorch FP32 LayerNorm dies silently past 3.4×10^38

A practitioner trained a weight-shared recursive Transformer, a design that applies the same block repeatedly, so hidden-state magnitudes accumulate with depth. At the 200M-token checkpoint, the hidden state just before the final LayerNorm reached a sum of squared deviations of 2.70×10^38, close to FP32's 3.4×10^38 ceiling, and rsqrt of the variance fell to 7.7×10^-19, effectively zero.

The minimal CPU script isolates the mechanism. Because LayerNorm sums the squared deviations across the hidden dimension, a single element's square can be only about 1/65 of the FP32 maximum while the row total already overflows. In the sweep, rows collapsed exactly when that row sum exceeded FP32's max, with zero mismatches across 12,248 rows. The CE during collapse sat near ln(32000)=10.37, a finite value that looks plausible in a training log.

The reported fix was to compute only the final LayerNorm in FP64 while keeping input and output dtypes unchanged. In the real model, collapse occurrences went from 6 to 0, usable recursion depth rose from 8 to 15, and validation CE at 200M tokens fell from 10.673 to 9.806, though the author notes other changes were also included. Applying FP64 to every layer was not benchmarked, and the author notes values could still grow beyond FP64's range.