30M-param LLM trained from scratch reveals scaling assumptions were wrong

The researcher built a decoder-only transformer from the ground up to understand how language models work by implementing and testing each piece by hand. The model architecture is standard—embeddings, multi-head causal self-attention with heads vectorized into a single batched matrix multiplication, and position-wise feed-forward networks with GeLU activation and d_ff = 4 × hidden_size—but every component is written in raw PyTorch tensor operations. The baseline configuration (configs/small.yaml) uses a hidden size of 512, 8 layers, 8 attention heads, a vocabulary of 8,000 tokens, and a sequence length of 256, yielding 33.6 million parameters. Training ran for 80,000 steps on the TinyStories dataset using AdamW with a linear warmup of 2,000 steps followed by cosine decay to 10% of peak learning rate, achieving a throughput of ~42,000 tokens per second with mixed-precision (fp16) training and batch size 32 on a single free Kaggle T4 GPU over approximately 4.5 hours.

A series of controlled ablation studies isolated the effect of specific design choices. The normalization ablation compared hand-written RMSNorm (which divides each token vector by its root-mean-square and applies a learned scale) against LayerNorm (which adds mean-subtraction and a shift parameter). Both 22k-step training runs produced quality ties (equivalent validation loss), but LayerNorm was ~33% faster due to the unfused RMSNorm implementation. The warmup-length ablation tested 50, 500, and 2,000 warmup steps and found negligible differences; even a stress variant with 10× learning rate and no gradient clipping did not diverge, confirming that AdamW dampens the importance of warmup at this scale. The learning-rate sweep (3e-4, 1e-3, 3e-3) identified 1e-3 as optimal, cutting validation loss by 0.042 compared to the original 3e-4 baseline, suggesting that peak learning rate is the real training lever.

The scaling study fit five models across a ladder from 1.8M to 56.6M non-embedding parameters to a power law L ≈ 1.16 + 0.78·N^−0.276 with R² 0.9975. A diagnostic run at 80k steps showed the floor was compute-limited rather than a data ceiling—contradicting the conventional assumption that small models hit an immovable architectural limit. The sample output from the prompt "Once upon a time" generated: "Once upon a time, there was a little girl named Lily. She loved to play in the garden with her toys. One day, she found a shiny rock and she wanted to keep it safe. She put it in her pocket and went to play with her toys. … She remembered the shiny rock in her pocket and thought it might make a magic wand." The model produces fluent, coherent TinyStories-style narratives with consistent characters and clear arcs, with only residual small-model artifacts (occasional repetition or mild logic drift) at this scale.

Evaluation used a custom TinyStories rubric with Llama-3.3-70B as an LLM judge, and comparison against GPT-2. A key finding was that specialization beats scale on the target distribution—the 30M model tuned for TinyStories outperformed larger, more general models when evaluated on that specific task. Interpretability analysis identified previous-token and attention-sink heads at both model sizes, with induction heads only weakly emerging. The project is fully reproducible: all code, versioned configs, shape and gradient-flow tests, and instructions for running on Kaggle are published in the GitHub repository, and the tokenized dataset is available as a private Kaggle Dataset so that the full 80k-step training can be repeated on free GPU.

The project stands out as a deliberate, hands-on deep-dive into transformer mechanics rather than a benchmark chase. By implementing every component—embeddings, multi-head causal self-attention, feed-forward networks, and the training loop—directly on PyTorch tensor operations (no high-level model libraries), the researcher gained direct visibility into which design choices actually matter at scale. The ablations were carefully scoped: RMSNorm vs LayerNorm produced a quality tie (LayerNorm ~33% faster due to unfused RMSNorm), warmup length proved negligible (AdamW makes it redundant), and learning rate was identified as the primary lever (1e-3 cutting validation loss by 0.042 vs the original 3e-4).

The most significant claim is the reinterpretation of the scaling "floor." Standard wisdom suggests small models plateau quickly and that scaling is the only path forward. The power-law fit (R² 0.9975) and the 80k diagnostic instead suggest the plateau observed at this compute budget is an artifact of insufficient training, not a model-size ceiling. When the researcher evaluated the model on its native distribution (TinyStories rubric via Llama-3.3-70B as judge), specialization to that distribution outperformed scale—a practical finding that smaller, domain-tuned models can be more useful than larger general ones when the task is well-defined.

The reproducibility constraints are deliberately tight: the code ships with versioned YAML configs, fixed random seeds, shape and gradient-flow tests in each module, and paths configured to run identically on Kaggle (where the model was trained) or locally. The tokenized dataset and tokenizer are published as a private Kaggle Dataset, so the full 80k-step training is repeatable on free GPU.

No comments yet. Be the first to share your thoughts!