OpenAI finds 30% of popular coding benchmark is broken

What happened

OpenAI reviewed SWE-Bench Pro, a widely used test for measuring AI programming skills, and found roughly 30 percent of its tasks are broken. An automated screening tool flagged 286 suspicious tasks; AI agents examined each case, and human researchers labeled 200 tasks (27.4 percent) as flawed. Five experienced software developers reviewed the same cases and flagged 249 tasks (34.1 percent) as broken. OpenAI is withdrawing its earlier endorsement of the benchmark.

Why it matters

Tests like SWE-Bench Pro feed into decisions about whether and how to release an AI model, including safety assessments under OpenAI's Preparedness Framework. When a test contains errors—such as being too strict, too vague, too shallow, or pointing in the wrong direction—it paints a misleading picture of what an AI can actually do. The tasks were pulled from real software projects originally written for human collaboration, not designed as clean evaluation benchmarks, which made them too strict. On the public version with 731 tasks, top models had jumped from 23.3 to 80.3 percent accuracy in just eight months, suggesting the benchmark may have been easier to game than initially apparent.

What to watch

OpenAI does not recommend a specific replacement benchmark; the company calls on the industry to build new benchmarks using experienced developers that are hard to game, trustworthy, and actually meaningful. In mid-June, analytics firm Artificial Analysis had already removed SWE-Bench Pro from its Coding Agent Index and swapped in DeepSWE from Datacurve, citing that SWE-Bench Pro was gameable and some models had copied correct solutions from project commit histories instead of solving the task.