Peri-LN posts best scores at 400M to 3.2B

The paper, authored by Jeonghoon Kim and colleagues from NAVER Cloud and KAIST, does not introduce a brand-new layout. Peri-LN was already used in Gemma and OLMo family models, but it had no unified name or published analysis of its properties. The researchers gave it a name — from the prefix peri- meaning surrounding, because normalization encloses each sub-layer front and back — and set out to study it systematically.

Their experiments trained Transformers at 400M, 1.5B, and 3.2B parameters, each on 30B tokens across 5 seeds, using mainly RMSNorm. Across all sizes, Peri-LN achieved the lowest evaluation loss and highest average benchmark score. The paper also notes that Pre-LN diverged frequently early in training regardless of seed (those diverged checkpoints were excluded from the tables), that Peri-LN showed small standard deviation in scores across seeds, and that at 3.2B Pre-LN's intermediate outputs exceeded FP16's maximum value at just 0.5B tokens into training.

Their analysis explains why: in Peri-LN, the residual path adds only normalized outputs, so variance grows roughly linearly rather than exponentially, and the output-side normalization keeps the FFN second-layer gradient norm bounded even when the first-layer output h becomes large (Proposition 3.1). This matters for practitioners because it points to a concrete training layout that could make large-model training more stable without relying on the assumption that Pre-LN is inherently safe.