Researcher tested 11 AI models grading each other's predictions — found most are unreliable judges of AI quality

A researcher asked 11 different AI models (like Claude, GPT-4, Gemini) to evaluate and score predictions made by other AI systems, then compared their grades to human expert judgment. The finding: most AI models were inconsistent or inaccurate when asked to judge other AI work — they disagreed with human experts and with each other on which predictions were actually good.

The core problem is that AI models lack a stable, objective standard for evaluation. When humans grade AI output, they use consistent criteria (accuracy, clarity, relevance). But when AI grades AI, the model may apply different logic each time or weight factors in ways that don't match real quality, especially for nuanced or subjective tasks.

This matters for anyone building or buying AI tools: if you're using an AI system to automatically filter, rank, or approve other AI output (for customer service, content moderation, or decision-making), you can't assume the AI judge is reliable. You need human spot-checks or explicit scoring rules, not just AI-to-AI evaluation.