Claude Fable 5 tops GPT-5.6 models in physics-based AI test

JuliaHub, which ships Dyad (an AI agent platform for modeling and simulation workflows), conducted an independent evaluation of four frontier AI models on five sealed physics problems to determine which handles engineering work best. The study pinned the Dyad harness, the problems, and all configuration parameters—reasoning effort (xhigh), context window (1M tokens), and token budget (128k)—while varying only the model. Across three trials on each of four core problems and one long-horizon trial on the hardest fifth problem, the company ran 52 graded trials total and scored every trial against sealed ground truth.

The results: Claude Fable 5 posted a weighted difficulty score of 0.889 at a cost of $9.60 per trial (16.1 minutes average runtime); GPT-5.6-sol scored 0.814 at $1.74 per trial (13.4 minutes); GPT-5.6-terra scored 0.786 at $1.25 per trial (12.6 minutes); GPT-5.6-luna scored 0.727 at $3.26 per trial (25.0 minutes). Fable was the only model to sweep all twelve trials on the four core problems. The fifth problem—NASA's HL-20 lifting body on approach—no model solved completely, but Fable scored highest among the graded tests where external reference existed.

The study analyzed transcripts behind every trial to uncover each model's work style. Fable verifies by constructing examples that fail: on the relativistic dynamics problem, it swept solver tolerances across three decades, checked fields against independent reimplementations at 200 points, and deliberately sign-flipped a component to prove the check could catch errors. It made only twenty-eight tool calls per trial because it tried to break every solution before committing it. Sol specifies more than the task asks—it built independent oracles and closed checks to high precision, but that same instinct caused its slip on the relativistic problem: it renormalized the state to match the coordinate velocity rather than the physical z-component, handing the particle 27% excess longitudinal momentum, and its own consistency check blessed the misreading because every downstream check was self-consistent and wrong. Luna iterates where it should investigate: it carried the least configuration into runs and reads more of the problem than anyone else, which served on easier problems but turned to churn when physics pushed back—68 tool calls on one constrained-consistency trial with a committed solution still 30% off. Terra economizes: it spends the smallest share of clock on derivation and testing and the largest on editing, getting away with it by producing correct formulations at the cheapest and fastest cost, but its slip on the steady-state linearization problem reveals the pattern—terra rewrote the required simulation horizon (from 5.0 to 3.0) and validated its reduction against a reference it gave the same truncated horizon, blinding its own check.

On the HL-20 frontier problem—a complete flight vehicle with six-degree-of-freedom rigid-body dynamics, aerodynamics from 170 wind-tunnel tables, a standard atmosphere, and control-surface logic from two NASA technical memos, graded on eight flight scenarios—the transcripts show how each model approached the hard work. Terra shipped a flight vehicle without reading the flight manual: its one attempt to extract the NASA memos failed at minute 3.5 and never retried, so it invented the control mixer, running the full flight four times without looking past the return code, with a control surface settling eleven degrees from spec, unremarked. Sol read everything and verified the parts: it extracted both memos by minute three, built aerodynamics as an interpreter over the supplied tables, and locked them against the 24 check cases NASA embedded in the data file—all passing by minute 27—but never verified a trajectory, with its dutch-roll run ending below sea level in a 48-degree dive, concluded with "the rudder's behavior seems correct." Luna never got the airplane to fly: it fought one compile error twenty-three times across eighty minutes, then resolved it by rewiring the vehicle's vertical aerodynamic force and pitch torque to zero, shipping scenarios it only ran for a tenth of a second, with the attitude quaternion never moving. Fable spent a third of its run reading before it wrote a line—spec, tables, both memos, rendering PDF pages as images to settle one ambiguous constant—then wrote sixteen files that compiled on the first try, ran all eight scenarios, and closed with a falsification battery against every oracle it could find.

The verdict came down to tradeoffs: on demonstrated capability, Fable posts the highest difficulty-weighted score and is the only model to sweep all twelve core-problem trials, but charges three to eight times more per trial. GPT-5.6-sol is second on merit at a fifth of Fable's price and built some of the study's best independent verification. Terra is the cheapest and fastest route to a credible score, trimming margins to get there. Luna trails on every axis. A companion study held the model constant (Fable) and varied the harness, running the same frontier model through Dyad versus Claude Code with the same tailored documentation: the stock agent scored 0.533, Dyad scored 0.899—a 0.366-point gap, more than double the 0.162 separating the best and worst frontier models in the main experiment. This finding suggests that the agent framework and verification tooling matter more than model choice alone.

JuliaHub's evaluation addresses a critical gap in how frontier AI models are assessed for engineering work. Vendor self-reported benchmarks face inherent conflicts of interest—the publisher has every incentive to tune for the score it announces. JuliaHub's incentives run the opposite direction: the company ships Dyad with multiple agent backends across vendors and wins when users get the best experience, whichever model delivers it. This alignment makes their findings credible where vendor claims are not.

The study design isolates model performance by holding every other variable constant: the Dyad AI agent harness, the five sealed problems, the reasoning effort level (xhigh), the context window (1M tokens), and the token budget (128k). Only the model changes. The evaluation also grounds scoring in ground truth rather than code quality—a crucial distinction in physical AI. A solution can compile, pass internal checks, and still encode impossible physics. The study's hardest problem, NASA's HL-20 lifting body, exemplifies this: it required reading two technical memos, extracting 170 wind-tunnel tables, implementing six-degree-of-freedom rigid-body dynamics, and running eight flight scenarios. No model solved it completely, but the transcripts reveal how each failed and why.

The inverse experiment—holding the model constant and varying the harness—delivers a striking finding: the framework matters more than the model choice. This suggests that businesses evaluating frontier models for engineering work should weight the agent infrastructure and verification tools as heavily as raw model capability. The cost-performance tradeoff also matters: Fable's higher cost reflects its deep verification strategy, while GPT-5.6-sol offers a middle path, and terra prioritizes speed and cost at the expense of margin. For teams where correctness is non-negotiable, Fable's conviction justifies the expense; for those optimizing value, sol presents a credible alternative.

No comments yet. Be the first to share your thoughts!