Study reveals three distinct phenomena in transformer weight matrix spectra during pretraining: transient compression waves, persistent spectral gradients, and asymmetry between query/key and value projections.

Researchers tracked full singular value decompositions of every weight matrix at 25-step intervals across three model scales (30M–285M parameters) and identified three spectral phenomena: transient compression waves that propagate from early to late layers, persistent power-law exponents that form a non-monotonic depth gradient, and asymmetric compression patterns where value/output projections compress uniformly while query/key projections carry full depth-dependent dynamics.

The study formalizes the findings as a two-timescale dynamical model with derived scaling laws (Δα ∝ L^0.26, R² = 0.99) and validates results on nine models across three families (custom, GPT-2, Pythia; 30M–1B parameters; 8–36 layers), showing that the power-law exponent predicts layer importance (ρ = 0.69–0.84, p < 0.02).

Spectral-guided pruning outperforms Last-N heuristics by 1.1×–3.6× across seven models in two families (GPT-2 124M–774M, Pythia 160M–1B), with worst-vs-best gaps up to 23.7×, confirming that spectral structure plays a causal role in determining which layers matter most.