GPT-6 Astra triples Claude Fable in Andon Labs tests

Andon Labs, which runs all its evaluations internally to prevent model-makers from tuning to the test, put GPT-6 Astra through two very different agent benchmarks. In Vending-Bench, models had to negotiate supplier deals, set retail prices and grow a bank balance over a simulated year. Astra's advantage showed up most clearly in procurement: where Fable 5.1 let its average Coca-Cola purchase price drift from $1.17 to $2.21, Astra held firm, including one case where it bargained a $226.32 quote down to $108. Fable 5.1 also lost $14,331 to prepayments made to suppliers that had already shut down, even after writing a rule to only pay after written confirmation — and then breaking that rule days later.

On Drone-Bench, where models write code for a cheap DJI Tello EDU to navigate an office and follow a specific person, Astra became the first model whose best attempts beat the human-AI reference code on all five subtasks, including 3D reconstruction, which had remained unsolved. Its pipeline combined COLMAP and DA3 with added depth filtering. But the benchmark's structure — ten runs per task and up to ten code submissions each — rewards best-case attempts rather than consistency, and Astra's one-in-ten success rate on reconstruction underscores that distinction.

Andon Labs argues that the public and lawmakers should understand these capabilities before AI-powered drones reach superhuman navigation skills. The practical test is whether the reliability gap closes: best-case wins on each subtask do not yet translate into a dependable end-to-end system, and the team's own projection of a single-attempt solution by Q1 2027 hinges on progress continuing at the pace of the past two years. For businesses weighing autonomous agents in procurement or logistics, Astra's behavior in the simulated vending business — consistent negotiation, no identified losses from supplier closures, no lying observed across three arena games — is the more immediately relevant finding, though Andon Labs cautions that benchmark behavior does not automatically transfer to other situations.