OpenAI revised GPT-6 Astra benchmark scores after launch. Some numbers favored Astra, while Anthropic's dropped.
OpenAI says fixes ensure accurate comparisons. Critics question the practice of re-running evals.
The confusion could affect customer trust and OpenAI's future IPO.
What happened
OpenAI changed several evaluation metrics for its GPT-6 Astra model after first publishing a blog post on Sept. 3. The numbers on updated versions showed Astra performing better, while Anthropic's scores got worse.
Why it matters
The changes highlight concerns about benchmark reliability and possible manipulation, as these scores help AI companies win customers and investors. OpenAI said it made fixes to ensure numbers reflect the best estimate of model performance.
What to watch
The hallucination rate was halved to 2% for Astra then reverted to 4.2%, while Sol's ExploitBench score rose to 11.5% and may be reverted. OpenAI faces scrutiny over its evaluation practices ahead of a possible 2027 IPO.
Ask the AI about this article →
The benchmark adjustments come as OpenAI and rivals race to lead in AI, where eval scores serve as a public scorecard. The changes, including halving Astra's hallucination rate to 2% and boosting Sol's ExploitBench score, could give OpenAI a marketing edge. But experts like Stanford researchers note that re-running evals is a known practice, and the system card provides few details. This raises questions about how much trust to place in these numbers. The confusion could complicate customers' model choices and muddy OpenAI's narrative of having the best models, especially ahead of a potential 2027 IPO.
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
Anthropic PBC used its Claude AI to create a computer-verifiable version of Andrew Wiles's 1995 proof of Ferma…
BEXCO general manager Tom Choi argues that South Korea's safety-tech sector, filled with AI cameras, robots, d…
Topco is targeting optical communications and cooling technologies as part of its strategy, reflecting a shift…
Microsoft announced plans to restructure its financial reporting and segment disclosures from fiscal 2027, con…
ChronoScale said on August 27 that it plans with Microsoft to deploy 50 megawatts of AI infrastructure in Nort…
An early user on Hacker News says GPT-6 Astra feels too aligned out of the gate, with overly legalistic interp…
