New AI models show major gains on single-pass reasoning evals

The Second Look Fellowship undertook a replication of single-forward-pass evaluation experiments originally conducted by Greenblatt in 2025 and 2026. Single-forward-pass evals constrain models to produce answers in a single computational pass—a realistic scenario for latency-sensitive applications—rather than allowing chain-of-thought reasoning that would require multiple steps.

The team first replicated the original experiments on Opus 4.5, one of the baseline models from the prior work. Their evaluations confirmed the trends and quantitative values described in Greenblatt's posts, validating their experimental setup. They then extended the evaluation to three newer models: Claude Fable 5, Opus 5, and GPT-5.6-Sol.

The newer models showed substantial performance gains on some tasks. Claude Fable 5 achieved 87.6% accuracy on Gen-Arithmetic when given 10 problem repeats, a significant jump from the previous state-of-the-art around 60%. GPT-5.6-Sol experienced pronounced uplift from two techniques—filler tokens (extra text inserted to influence reasoning) and problem repeats (presenting the same problem multiple times). Across all 4 datasets tested, these techniques improved GPT-5.6-Sol's performance, and on the 3-hop reasoning task specifically, they roughly doubled performance relative to the baseline with no chain-of-thought.

The research team plans to release open-source tooling to enable other researchers to apply single-forward-pass evaluation elicitation methods and to conduct more comprehensive replications of previous work in subsequent posts.