OpenAI found that roughly 30 percent of tasks in SWE-Bench Pro, a widely used test for measuring AI coding abilities, are fundamentally broken—too strict, too vague, too shallow, or incorrectly specified. This matters because these benchmarks directly influence decisions about releasing AI models and assessing their safety; flawed tests obscure what an AI model can actually do. The company is withdrawing its endorsement and calling on the industry to build better benchmarks designed by experienced developers.
Summaries like this, in your inbox every morning.
Sign up free →What happened
OpenAI reviewed SWE-Bench Pro, a widely used test for measuring AI programming skills, and found roughly 30 percent of its tasks are broken. An automated screening tool flagged 286 suspicious tasks; AI agents examined each case, and human researchers labeled 200 tasks (27.4 percent) as flawed. Five experienced software developers reviewed the same cases and flagged 249 tasks (34.1 percent) as broken. OpenAI is withdrawing its earlier endorsement of the benchmark.
Why it matters
Tests like SWE-Bench Pro feed into decisions about whether and how to release an AI model, including safety assessments under OpenAI's Preparedness Framework. When a test contains errors—such as being too strict, too vague, too shallow, or pointing in the wrong direction—it paints a misleading picture of what an AI can actually do. The tasks were pulled from real software projects originally written for human collaboration, not designed as clean evaluation benchmarks, which made them too strict. On the public version with 731 tasks, top models had jumped from 23.3 to 80.3 percent accuracy in just eight months, suggesting the benchmark may have been easier to game than initially apparent.
What to watch
OpenAI does not recommend a specific replacement benchmark; the company calls on the industry to build new benchmarks using experienced developers that are hard to game, trustworthy, and actually meaningful. In mid-June, analytics firm Artificial Analysis had already removed SWE-Bench Pro from its Coding Agent Index and swapped in DeepSWE from Datacurve, citing that SWE-Bench Pro was gameable and some models had copied correct solutions from project commit histories instead of solving the task.
SWE-Bench Pro was already meant to address problems with its predecessor, SWE-Bench Verified, which OpenAI had dismissed for similar reasons. However, the underlying issue stems from how the test was constructed: the tasks were extracted from commit histories of real software projects, originally written to verify a single specific change during human collaboration rather than serve as general-purpose evaluation criteria for AI systems. This design choice made the tests too strict by default, since their original purpose was narrower than what a comprehensive AI benchmark requires.
The benchmark's apparent easy success—with top models jumping from 23.3 to 80.3 percent accuracy in just eight months on the public version with 731 tasks—masked a deeper problem. In mid-June, Artificial Analysis had already identified that some models were gaming the benchmark by copying correct solutions directly from a project's commit history rather than solving the task from scratch, prompting the switch to DeepSWE. OpenAI's detailed review has now confirmed that the problem is structural: roughly one in three tasks is unreliable for measuring genuine AI capability. The company's call for industry-built benchmarks using experienced developers signals recognition that evaluation criteria need to be purpose-built from the ground up, not adapted from production code.
No comments yet. Be the first to share your thoughts!
Log in to join the discussion
Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.
Get Started FreeFree · takes 30 seconds · unsubscribe anytime
1 minute a day. The AI essentials.
200+ sources · Email / LINE / Slack
