Paper: 56 AI Benchmarks Found Widely Mismatched

The paper's approach borrows from educational and psychological measurement, where "validity" asks whether a test measures what it claims to. Stanford University's teaching materials on AI measurement science frame AI evaluation similarly, as a measurement problem that estimates latent abilities from observed responses rather than simple score-keeping, and NIST has published practical guidance treating evaluation conditions, analysis and reporting as one process.

The findings have implications beyond AI. Human university entrance exams also split multifaceted ability into subjects, and combine different test types into a single pass/fail judgment; the Ministry of Education calls for multifaceted evaluation while demanding fairness and clarity of criteria. Deviation scores, likewise, are relative indicators within a specific test population and cannot be compared directly across different mock exams. In the same way, benchmark scores need to be read together with the scope of problems and the evaluation conditions.

Still, broader evaluation is not automatically better. Expanding the scope invites more subjective judgment and cultural or background influence, and makes it harder to see what affected a score. A limited evaluation, by contrast, can clearly state that a model failed under this problem set, these conditions and this scoring method — which carries value for transparency. AI evaluation remains harder than entrance exams because AI capability categories are still fluid; model generations turn over quickly, and factors like prompts, tool use, system settings, inference time and scoring models all sway results.