Schema harness hits 99% on ARC-AGI test with Claude Opus 4.8

Schema is a harness—a wrapper or orchestration layer—that improves performance on the ARC-AGI reasoning benchmark by refining how models process observations, test predictions, and revise plans. The system was introduced alongside claims of 99% accuracy on the ARC-AGI-3 Public set when paired with Claude Opus 4.8 and Fable 5, and 95.35% accuracy with GPT-5.6 Sol.

Crucially, Schema does not retrain or modify the weights of the underlying models. Instead, it improves the process around them. Specifically, it changes how observations are converted into a working model of the game, how predictions are tested against the interaction history, and how plans are executed and revised. This approach suggests that orchestration—the way a system coordinates inference, feedback, and iteration—can yield substantial gains on reasoning tasks.

The harness uses a fixed fallback rule to combine model strengths: Opus 4.8 and Sol xhigh run first; if a game scores below 80, it reruns with Fable 5 and Sol max, respectively, and retains the higher per-game score. This dual-run strategy allows the system to benefit from different model strengths on different problems without requiring a single larger model.

The ARC Prize president responded on social media with "Looks cool - need to dig into it," indicating interest from the benchmark's leadership while also suggesting that the result warrants deeper scrutiny. Technical details are available on the Schema project's website (schema-harness.github.io).

Schema represents a shift in how reasoning benchmarks can be approached: rather than pushing for larger or more capable base models, the team focused on improving the process layer that surrounds inference. The 99% result on ARC-AGI-3 is particularly noteworthy because it demonstrates that significant accuracy gains are possible through better orchestration—observation parsing, history-based prediction testing, and plan revision—without modifying the underlying model weights. The use of a fallback rule (rerunning games that score below 80 with different model variants and keeping the higher score) is pragmatic but also reveals that performance gains come partly from model diversity and partly from the harness's process improvements.

The ARC Prize president's cautious but positive response ("Looks cool - need to dig into it") suggests the result is being taken seriously within the benchmark community, though the need for deeper investigation indicates the mechanism and generalizability of the gains may not yet be fully established.

No comments yet. Be the first to share your thoughts!