What did the Apple researchers actually measure?

They generated text with four LLMs for three long-form writing tasks, collected human labels on which of two responses was better, collected LLM judge pairwise-ranking annotations, and compared the human and judge annotations.

Does a higher MMLU score mean a better writing judge?

No. The researchers found judge quality correlates weakly with general LLM ability measured using MMLU, with Pearson's ρ ≤ 0.2.

Which judging method worked better?

LLM evaluators were more consistent and performant using paired comparisons rather than scoring responses independently.

Large Language ModelsAI Safety & AlignmentTop Companies' AI MovesTop Companies AI10m agoNov 12, 2026, 06:30 JST

Apple finds LLM judges weak on style: MMLU ρ ≤ 0.2

Send on LINE

3 Key Points

What happened
Apple researchers compared human and LLM judge annotations across four LLMs and three long-form writing tasks, finding judge quality correlates weakly with MMLU (Pearson's ρ ≤ 0.2) and varies by task and evaluation criteria.
Why it matters
The researchers also found that the strongest LLMs do not most reliably reproduce human evaluations, which they hypothesize relates to exaggerated sensitivity to subtle details that human annotators and weaker LLMs ignore — suggesting teams cannot assume a bigger model makes a better judge.

WHO IT HITSTeams building or buying AI writing tools that rely on LLM judges to score personalized style will need to validate judge performance per task and criteria rather than assuming stronger models judge better, since the Apple researchers found the strongest LLMs were not the most reliable at reproducing human evaluations.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Get these free every morning

Context & Analysis

The paper sits at the intersection of two trends Apple's researchers describe in related work: LLMs acting as writing agents that infer user preferences, and the difficulty of evaluating whether generated text actually matches a stated preference like "upbeat." Their experiment separates the writing task, the generator-judge pairing, the evaluation objective, and general reasoning ability, then checks each against human labels. The finding that judge quality tracks only weakly with MMLU suggests general model benchmarks are a poor proxy for judging personalized style. The paired-comparison result adds a methodological wrinkle: how the judge is asked matters, not just which model is asked. The most counterintuitive result is that the strongest LLMs were not the most faithful to human labels, which the authors hypothesize stems from exaggerated sensitivity to subtle details that human annotators and weaker LLMs ignore — a reminder that in evaluation, more capability may not mean more agreement with the people whose preferences are being modeled.

FAQ

What did the Apple researchers actually measure?: They generated text with four LLMs for three long-form writing tasks, collected human labels on which of two responses was better, collected LLM judge pairwise-ranking annotations, and compared the human and judge annotations.
Does a higher MMLU score mean a better writing judge?: No. The researchers found judge quality correlates weakly with general LLM ability measured using MMLU, with Pearson's ρ ≤ 0.2.
Which judging method worked better?: LLM evaluators were more consistent and performant using paired comparisons rather than scoring responses independently.

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.