KAIST study: AI reasoning steps separable inside models

The study, conducted by researchers at South Korea's KAIST and Naver AI Lab, set out to test whether the distinct reasoning steps a language model shows in its text output can also be found in its internal states. They defined eight recurring reasoning operations, including extraction, decomposition, formula recall, deduction, and computation, and had three models solve math problems. The solution paths were split into segments and labeled using GPT-5.

The different operations could be reliably told apart in the models' internal representations across all three models, with the separation peaking in the middle layers. A classifier looking only at the tokens used performed worse, and position within the solution path didn't explain the effect either. Even on incorrectly solved problems, the type of step the model was performing stayed identifiable internally. The effect replicated with Llama-3-8B, and for Qwen3-8B the trained classifiers transferred to GPQA-Diamond and MATH-500.

The findings matter for AI safety because reading the chain of thought is one of the few oversight tools available, yet Anthropic showed models only disclose the hints they used in 25 to 39 percent of cases. Whether this method can be used to catch errors or steer a model mid-generation remains an open question, and the experiments are limited to math tasks and a handful of models. The outcome hinges on whether future work can extend the approach beyond math and into practical oversight.