AIToday
Large Language ModelsAI Safety & AlignmentAI WeeklyPublished: Nov 19, 2026, 16:00 JST

Audit: 22 of 40 LLM reports reuse benchmark scores

Send on LINE
Audit: 22 of 40 LLM reports reuse benchmark scores

3 Key Points

  1. What happened

    Bahaeddin Türkoğlu audited 40 LLM technical reports and found 22 took some or all benchmark comparison scores from other papers instead of re-running them. Only 1 report disclosed data mixture, token count, and compute together.

  2. Why it matters

    Capability rankings built on scores from different evaluation setups lack a shared baseline, so the comparisons practitioners use to choose and fund models may not be verifiable, per the paper.

WHO IT HITSAI procurement and research teams that pick models from vendor technical reports may be relying on benchmark numbers produced under different setups, and the paper suggests those comparisons are often unverifiable without the original training runs.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The audit's two parts target different links in the same chain. The report analysis looks at what labs claim about their models, while the code review looks at what those models are actually built from. Of 73 architectural or training novelty claims, only 19 included a controlled ablation, and just 4 reports described more than one training run. Only 8 included any third-party evaluation.

The author is careful not to overstate the case: an untested claim may still be true, and copied scores are sometimes unavoidable when a comparison model cannot be re-run. The proposed remedy is procedural rather than technical — seven minimum reporting rules — with code and data released publicly so others can check the work.

FAQ
How many of the audited reports re-ran their benchmark comparisons?
22 of the 40 reports took some or all benchmark comparison scores from other papers rather than re-running them. Eight ran comparison models under a single evaluation setup.
What did the audit of Hugging Face models find?
Of 48 models added to Hugging Face transformers in 2025, 39 share at least two of their three core building blocks with an earlier model, most often Llama.
Does the paper say copied scores are always wrong?
No. The author notes an untested claim may still be true, and copied scores are sometimes unavoidable when a comparison model cannot be re-run.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleOmron to buy OnLogic for $675 million by January 2027