Apple: LLM translation raters beat humans at times, still unreliable

Apple's researchers set out to test a widely held assumption: that large language models can judge machine translation quality more cheaply and efficiently than people. They compared LLM agreement with human annotators on two established schemes, Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA), across a long-context test set of 70 language pairs and the WMT23 and WMT25 datasets.

The results were mixed. In some tasks, LLMs agreed with human annotators more than human annotators agreed with each other. But that agreement varied substantially across annotation schemes, language pairs and domains, and both approaches remained unreliable for most settings.

The paper also pinpoints where each side falls short: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. The authors suggest human-LLM collaborative annotation pipelines could address the reliability issues they identified.