Researchers explain why entropy signals reliably predict reasoning success in large language models through the Stepwise Informativeness Assumption framework

Study resolves a key puzzle in LLM research: why internal entropy dynamics correlate strongly with correct external answers

Introduces the Stepwise Informativeness Assumption (SIA), which posits that reasoning prefixes accumulate answer-relevant information as generation progresses

SIA naturally emerges from maximum-likelihood optimization on human reasoning traces and is reinforced by standard fine-tuning and reinforcement-learning approaches

The framework explains how autoregressive models effectively reason by progressively gathering information about the true answer through intermediate steps