Researcher fixes flaw in how AI models are ranked — benchmark disagreements were caused by test data leaking into training

A researcher published findings showing that most popular LLM (large language model — AI that understands and generates text) benchmarks rank AI models inconsistently because test questions are accidentally included in the AI's training data. The fix: train the model first, then test it on completely separate data. This simple reordering revealed that some widely-praised models actually underperform competitors.

The problem was invisible because benchmark creators and AI companies were measuring performance on data the models had already seen during training — like studying for a test by memorizing the exact exam questions. When researchers prevented this data leakage, the rankings changed significantly, and some models dropped dramatically in their reported scores.

For anyone choosing an AI tool for work (writing, coding, research assistance), this means past benchmark comparisons you've read may have been misleading. You should distrust existing 'best LLM' rankings published before this fix was widely adopted — the actual performance differences between models may be much smaller or reversed from what was reported.