Why did OpenAI change the benchmark numbers?

OpenAI said the changes were fixes to ensure numbers represent the best estimate of available model performance. The company also noted that most evaluations have noise within a few percentage points.

What was the most notable change?

Astra's hallucination rate was halved to 2% in later versions of the blog, then reverted to 4.2%. Also, GPT-5.6 Sol's ExploitBench score rose from 5.5% to 11.5% and may be reverted.

AI Safety & AlignmentAI Business & IndustryFortune AISep 5, 2026Sep 5, 2026, 10:01 JST

OpenAI quietly tweaks GPT-6 Astra benchmark scores post-launch

Send on LINE

3 Key Points

What happened
OpenAI changed several evaluation metrics for its GPT-6 Astra model after first publishing a blog post on Sept. 3. The numbers on updated versions showed Astra performing better, while Anthropic's scores got worse.
Why it matters
The changes highlight concerns about benchmark reliability and possible manipulation, as these scores help AI companies win customers and investors. OpenAI said it made fixes to ensure numbers reflect the best estimate of model performance.
What to watch
The hallucination rate was halved to 2% for Astra then reverted to 4.2%, while Sol's ExploitBench score rose to 11.5% and may be reverted. OpenAI faces scrutiny over its evaluation practices ahead of a possible 2027 IPO.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

Context & Analysis

The benchmark adjustments come as OpenAI and rivals race to lead in AI, where eval scores serve as a public scorecard. The changes, including halving Astra's hallucination rate to 2% and boosting Sol's ExploitBench score, could give OpenAI a marketing edge. But experts like Stanford researchers note that re-running evals is a known practice, and the system card provides few details. This raises questions about how much trust to place in these numbers. The confusion could complicate customers' model choices and muddy OpenAI's narrative of having the best models, especially ahead of a potential 2027 IPO.

FAQ

Why did OpenAI change the benchmark numbers?: OpenAI said the changes were fixes to ensure numbers represent the best estimate of available model performance. The company also noted that most evaluations have noise within a few percentage points.
What was the most notable change?: Astra's hallucination rate was halved to 2% in later versions of the blog, then reverted to 4.2%. Also, GPT-5.6 Sol's ExploitBench score rose from 5.5% to 11.5% and may be reverted.

Fortune AIRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.