Co-Design Bench: GPT-6 Astra fully solves only 26.4% of real coding tasks

The new benchmark arrives as coding agents have saturated older tests. The authors point out that when models can run, check, and fix their own code, they hit scores above 99 percent on existing benchmarks, making differences between models hard to measure. Co-Design Bench deliberately selects harder real-world requests, and the resulting gap is stark: GPT-6 Astra fully solves only 26.4 percent of tasks, and just under 18 percent of repository tasks.

The study also shows that the harness—the program through which an agent reads, writes, and executes code—shapes results as much as the model. Qwen3.8-27B scored 75 or 56 percent on larger overhauls depending on the environment, and which of two models came out ahead also depended on the environment. The authors therefore recommend always reporting model and working environment together, which for companies selecting a coding agent means testing both as a pair.

The benchmark has limits: it evaluates one fixed request per task without follow-up questions or mid-task corrections, and the AI judges can make mistakes. This echoes earlier findings, such as METR's study showing that the actual acceptance rate of AI-generated code by project leads was roughly 24 percentage points below scores on SWE-bench Verified, highlighting how benchmark results can diverge from real-world usability.