Claude Fable 5 tops GPT-5.6 models in physics-based AI test

JuliaHub's evaluation addresses a critical gap in how frontier AI models are assessed for engineering work. Vendor self-reported benchmarks face inherent conflicts of interest—the publisher has every incentive to tune for the score it announces. JuliaHub's incentives run the opposite direction: the company ships Dyad with multiple agent backends across vendors and wins when users get the best experience, whichever model delivers it. This alignment makes their findings credible where vendor claims are not.

The study design isolates model performance by holding every other variable constant: the Dyad AI agent harness, the five sealed problems, the reasoning effort level (xhigh), the context window (1M tokens), and the token budget (128k). Only the model changes. The evaluation also grounds scoring in ground truth rather than code quality—a crucial distinction in physical AI. A solution can compile, pass internal checks, and still encode impossible physics. The study's hardest problem, NASA's HL-20 lifting body, exemplifies this: it required reading two technical memos, extracting 170 wind-tunnel tables, implementing six-degree-of-freedom rigid-body dynamics, and running eight flight scenarios. No model solved it completely, but the transcripts reveal how each failed and why.

The inverse experiment—holding the model constant and varying the harness—delivers a striking finding: the framework matters more than the model choice. This suggests that businesses evaluating frontier models for engineering work should weight the agent infrastructure and verification tools as heavily as raw model capability. The cost-performance tradeoff also matters: Fable's higher cost reflects its deep verification strategy, while GPT-5.6-sol offers a middle path, and terra prioritizes speed and cost at the expense of margin. For teams where correctness is non-negotiable, Fable's conviction justifies the expense; for those optimizing value, sol presents a credible alternative.