LongJudgeBench introduces comprehensive benchmark for evaluating LLM judges on long-form outputs, revealing substantial reliability gaps in current methods

Researchers introduced LongJudgeBench, a benchmark for evaluating LLM judges (AI systems that assess quality of text generated by other AIs) across diverse real-world scenarios and judging protocols, with code made available.

Long-form evaluation requires judges to assess document-level factors including overall organization, task-relevant coverage and depth, cross-section consistency, and scenario-specific quality criteria—more complex than short-form evaluation.

Results show current LLM judges remain unstable across scenarios; rubrics or references help but are not always sufficient to ensure reliable evaluation.