
What happened
Bahaeddin Türkoğlu audited 40 LLM technical reports and found 22 took some or all benchmark comparison scores from other papers instead of re-running them. Only 1 report disclosed data mixture, token count, and compute together.
Why it matters
Capability rankings built on scores from different evaluation setups lack a shared baseline, so the comparisons practitioners use to choose and fund models may not be verifiable, per the paper.
WHO IT HITSAI procurement and research teams that pick models from vendor technical reports may be relying on benchmark numbers produced under different setups, and the paper suggests those comparisons are often unverifiable without the original training runs.
Summaries like this, every morning.
The audit's two parts target different links in the same chain. The report analysis looks at what labs claim about their models, while the code review looks at what those models are actually built from. Of 73 architectural or training novelty claims, only 19 included a controlled ablation, and just 4 reports described more than one training run. Only 8 included any third-party evaluation.
The author is careful not to overstate the case: an untested claim may still be true, and copied scores are sometimes unavoidable when a comparison model cannot be re-run. The proposed remedy is procedural rather than technical — seven minimum reporting rules — with code and data released publicly so others can check the work.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
Superhuman, formerly Grammarly, added a personalized assistant to its Go multi-agent platform that works acros…
Alibaba CEO Eddie Wu said on the company's earnings call that the next Qwen AI model is being developed using…

Linq raised $90 million in a Series B led by Andreessen Horowitz at a $1 billion valuation, lifting total fund…

Anthropic's IPO is poised to raise billions, with Amazon and Google holding stakes valued around a combined $3…

OpenAI is rolling out Live Dictation globally on all ChatGPT plans across web, desktop, iOS and Android; it tr…

Google said Google AI Ultra (from 14,500円/month) now upgrades its included YouTube Premium from individual to…
