OpenAI finds ~30% of SWE-Bench Pro tasks broken

OpenAI's audit of SWE-Bench Pro follows its earlier criticism of SWE-bench Verified, another popular coding benchmark. The company had previously encouraged the community to switch to SWE-Bench Pro, but this new analysis reveals similar fundamental issues. These problems arise because benchmark tasks are sourced from real-world coding changes, where tests are written for human collaboration, not for evaluating AI models in isolation.

The findings highlight the difficulty of creating benchmarks that are both challenging and fair. OpenAI's use of AI agents to help audit the data shows a new approach to quality control, but human reviewers still found more broken tasks than the automated pipeline did. This suggests that even advanced tools may undercount issues, emphasizing the need for rigorous, multi-layered evaluation.

The stakes are high for OpenAI, as these benchmarks inform their deployment and safety decisions. If evaluations are flawed, they could misrepresent model capabilities and affect research priorities. The company's retraction of its earlier endorsement signals a need for the broader AI community to invest in more reliable evaluation methods, likely built with direct human involvement to ensure they truly reflect model abilities.