Imitative learning, not RL, remains LLMs' core source of capability

The researcher begins by acknowledging that reinforcement learning from verifiable rewards (RLVR) has become a prominent topic in LLM training research. However, the central thesis is that this visibility has created a distorted picture of how LLMs actually acquire their capabilities. The author identifies two distinct sources: imitative learning, which includes pretraining (exposing the model to vast amounts of text) and supervised fine-tuning (adjusting the model based on labeled examples), and reinforcement learning, which includes RLHF (reinforcement learning from human feedback), RLAIF (reinforcement learning from AI feedback), and RLVR. The author argues that when examining a fully trained LLM, imitative learning accounts for far more of the model's final capabilities than reinforcement learning does. To support this framing, the author references an earlier discussion of LLM pretraining, describing it as a process that 'magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work'—suggesting that pretraining contains capabilities and learning mechanisms that remain conceptually mysterious but empirically undeniable. The post positions itself as a corrective to the field's recent emphasis on RL techniques, arguing that researchers and practitioners should not lose sight of the foundational role played by imitative learning even as newer RL methods receive substantial research funding and media attention.

Reinforcement learning techniques—particularly newer approaches like RLVR—have captured significant attention in the LLM research community. The author acknowledges this shift is real and substantial, but argues it reflects a rhetorical or research-priority imbalance rather than the true source of LLM power. The core claim is that when one examines a fully trained LLM and asks which training phase contributed most to its observed capabilities, the answer is overwhelmingly the imitative-learning phase (pretraining on large text corpora and subsequent supervised fine-tuning on curated examples), not the RL phase that comes later. This distinction has practical implications: if imitative learning is indeed dominant, then scaling, data quality, and architectural improvements in that phase may yield larger capability gains than incremental RL refinements. The post also references the author's earlier work on how pretraining 'magically transmutes observations into behavior,' suggesting a view that the mechanisms underlying imitative learning in LLMs remain poorly understood but are nonetheless extraordinarily powerful.

No comments yet. Be the first to share your thoughts!