GPT-5.6 Sol triples ARC-AGI-3 score with two settings

OpenAI's experiment on ARC-AGI-3 reveals a common pitfall in AI evaluation: the harness, not the model, can be the bottleneck. GPT-5.6 Sol, which has solved open math problems and beaten games like Pokémon FireRed, initially scored only 13.3% on the public set. The default harness discarded the model's private reasoning after each action and employed a rolling truncation that dropped older observations, effectively forcing the model to relearn the game every turn. By switching to the Responses API with retained reasoning and compaction—settings that mirror production use in ChatGPT and Codex—the score jumped to 38.3% with 6x fewer output tokens.

This discrepancy underscores that benchmarks measure a bundle of choices—API settings, harness design, prompting—alongside model capability. The ARC-AGI-3 harness was intentionally generic for fairness, but it may not suit frontier models like GPT-5.6, which are trained to use private reasoning. OpenAI's recommendation that developers use the Responses API with reasoning and compaction could push the industry toward more representative evals, but this claim hinges on whether other models and harnesses benefit similarly.