RL for LLM reasoning is sparse policy selection, not learning

What happened

Researchers analyzed how reinforcement learning (RL) improves reasoning in large language models and found that RL does not teach new strategies—it redistributes probability mass over solutions the base model already contains. Token-level analysis across multiple model families and RL algorithms shows RL's benefit is concentrated at only 1–3% of token positions (high-entropy decision points where the model is uncertain), and the promoted token always lies within the base model's top-5 alternatives.

Why it matters

The findings reframe reasoning improvement as sparse policy selection rather than capability acquisition, suggesting the full RL optimization loop may be unnecessary. Because the base model's own entropy can identify the important positions without any RL-trained model, the correction is low-dimensional and representable in a tiny fraction of model parameters—opening a path to dramatically cheaper reasoning improvements.

What to watch

The researchers introduced ReasonMaxxer, an RL-free method that applies contrastive loss only at entropy-gated decision points using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matched or exceeded full RL performance while requiring only tens of problems and minutes of single-GPU training—a reduction in training cost of roughly three orders of magnitude.