OpenAI's GPT-5.6 Sol tops Opus 5 on ARC-AGI-3 with custom API settings

The dispute over GPT-5.6 Sol's ARC-AGI-3 score reflects a fundamental tension in AI benchmarking: the benchmark measures not just raw model capability but the entire system surrounding it. OpenAI's argument—that benchmarks inherently depend on technical setup—is factually sound, yet ARC-AGI-3 was explicitly designed to test pure model performance using a standardized approach to ensure fair cross-provider comparison.

The crux of the disagreement concerns whether the comparison itself was inequitable from the start. OpenAI contends that Anthropic may have used an older API with fewer features when ARC Prize ran its tests, giving Claude an unfair advantage in the official environment. ARC Prize co-founder François Chollet's response draws a careful distinction: general-purpose API settings that are publicly available to all users and were not developed specifically for the benchmark are acceptable; custom harnesses designed to exploit the benchmark format are not. This distinction permits OpenAI's use of Retained Reasoning and Compaction—features now available to all API users—while protecting the benchmark's integrity against purpose-built workarounds.

Chollet acknowledged ongoing collaboration with OpenAI "about how to best test their models, especially with regard to compaction," suggesting the two organizations are working toward a framework where technical differences are transparent and comparable. The unresolved question is whether future benchmarks will require all providers to use identical settings, or whether clearly disclosed provider-specific configurations will become standard practice.