Xin Liu paper finds 52.6% of untied gated checkpoints have near-zero embeddings

The finding stems from a survey of 291 open-weight LLM repositories, covering 280 distinct embedding tables. Among the 38 untied gated and hybrid checkpoints, 20 carried input embeddings with norms below 10−6, while weight-tied models were unaffected. This prevalence suggests that the defect is not an isolated case but a recurring pattern across checkpoints of these architectures. The mechanism is that near-zero token embeddings amplify backward gradients by more than 30 orders of magnitude, with individual entries exceeding 1.84 × 1019, leading gradient clipping to overflow and discard updates silently. In a Telugu continual pretraining run of Nemotron-3-Nano-4B, updates were discarded on approximately 20% of training steps, illustrating the practical impact. The authors propose two fixes, GroupClip and GateFloor, which in experiments up to 370 million parameters reduced problematic gradients by 28 orders of magnitude while maintaining perplexity comparable to the unpatched baseline. These results point to a straightforward mitigation path, though the prevalence figure underscores the value of auditing embedding norms before any continued pretraining on these architectures.