
What happened
OpenAI stopped reporting SWE-bench Verified scores in February, after reviewing the 138 problems AI models failed and finding 59.4% had serious flaws in grading tests or problem text.
Why it matters
A benchmark long used to compare programming ability can no longer reliably show whether higher scores mean real capability, since correct fixes could fail and top models may have seen the problems during training.
Summaries like this, every morning.
Benchmarks are meant to be a shared yardstick, but that yardstick wears out in two ways. The first is saturation: MMLU, a 57-subject multiple-choice test, went from 43.9% for GPT-3 to 86.4% for GPT-4 in 2023, and some models have since passed 90%. A separate study also found about 6.5% of MMLU's questions themselves were flawed. The response has been to build harder tests such as MMLU-Pro, GPQA, and Humanity's Last Exam, with roughly 2,500 questions.
The second problem is data contamination. When published test questions and answers end up in a model's training data, the score no longer measures the ability to solve unseen problems. SWE-bench Verified, a set of 500 expert-chosen problems, became a standard comparison for coding ability — which is why OpenAI's decision to stop reporting scores on it carries weight. It reviewed the 138 problems its AI failed and found 59.4% had serious flaws in the grading tests or problem text, so even a correct fix could be marked wrong. It also said frontier models reproduced human-written fixes on some problems, suggesting they may have seen them during training.
Benchmarks are like a car's advertised fuel economy: useful for comparison on the same basis, but real-world results vary with the road and how you drive. In AI, scores shift with how a prompt is written, how much computation a reasoning model is allowed, and the external tools or harness used. With few questions, a gap of a few points is a shaky basis for ranking models — so a benchmark leader is not automatically the best choice for a given job.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
AIworker published a free November 2026 report on structuring Microsoft 365 Copilot training for company-wide…

株式会社粋農 implemented MCP (Model Context Protocol, a common standard linking AI apps to external data and functio…

Anthropic struck a rare megadeal in AI infrastructure, committing $3.4 billion to take over orders for 1.7GW o…

Autobacs Seven introduced Google Workspace and Gemini across the group, shifting staff off macro-driven spread…

At Adobe MAX 2026, Adobe said it will support MCP in its main tools and offer its own Adobe Creative Agent sta…

In an October preprint, a researcher describes a candidate nonperturbative lattice definition of the Standard…
