AIToday
Large Language ModelsAI Coding AssistantsAI Watch (Impress)Published: Nov 16, 2026, 13:00 JST

OpenAI drops SWE-bench Verified scores after 59.4% flaw finding

Send on LINE
OpenAI drops SWE-bench Verified scores after 59.4% flaw finding

3 Key Points

  1. What happened

    OpenAI stopped reporting SWE-bench Verified scores in February, after reviewing the 138 problems AI models failed and finding 59.4% had serious flaws in grading tests or problem text.

  2. Why it matters

    A benchmark long used to compare programming ability can no longer reliably show whether higher scores mean real capability, since correct fixes could fail and top models may have seen the problems during training.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

Benchmarks are meant to be a shared yardstick, but that yardstick wears out in two ways. The first is saturation: MMLU, a 57-subject multiple-choice test, went from 43.9% for GPT-3 to 86.4% for GPT-4 in 2023, and some models have since passed 90%. A separate study also found about 6.5% of MMLU's questions themselves were flawed. The response has been to build harder tests such as MMLU-Pro, GPQA, and Humanity's Last Exam, with roughly 2,500 questions.

The second problem is data contamination. When published test questions and answers end up in a model's training data, the score no longer measures the ability to solve unseen problems. SWE-bench Verified, a set of 500 expert-chosen problems, became a standard comparison for coding ability — which is why OpenAI's decision to stop reporting scores on it carries weight. It reviewed the 138 problems its AI failed and found 59.4% had serious flaws in the grading tests or problem text, so even a correct fix could be marked wrong. It also said frontier models reproduced human-written fixes on some problems, suggesting they may have seen them during training.

Benchmarks are like a car's advertised fuel economy: useful for comparison on the same basis, but real-world results vary with the road and how you drive. In AI, scores shift with how a prompt is written, how much computation a reasoning model is allowed, and the external tools or harness used. With few questions, a gap of a few points is a shaky basis for ranking models — so a benchmark leader is not automatically the best choice for a given job.

FAQ
Why did OpenAI stop reporting SWE-bench Verified scores?
OpenAI reviewed the 138 problems its AI could not solve on SWE-bench Verified. It found 59.4% had serious flaws in the grading tests or problem statements, and that frontier models may have seen some problems during training.
What is SWE-bench Verified?
It is a benchmark made of 500 problems chosen by human experts, long used as a standard measure of how well AI models can fix real software bugs.
Why do AI benchmarks lose their usefulness over time?
Problems that were once hard become easy as models improve, so scores stop separating models — a saturation effect. MMLU, for example, went from 43.9% for GPT-3 to 86.4% for GPT-4 in 2023.
AI Watch (Impress)Read Original Article
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleAnthropic IPO on the 19th; PKSHA among partners