GitHub launches ReviewBench, an open AI code review benchmark

Agentic code review is increasingly used to inspect pull requests and catch issues before code ships, but teams have had a hard time measuring how well these AI reviewers actually work. Existing benchmarks often trade off label quality, coverage, and real-world representativeness, so GitHub built ReviewBench to combine all three. The benchmark analyzes 103.9 million GitHub pull requests to model language, repository size, and even pull request size, deliberately upweighting the reviewable middle and tail so tiny, single-file changes do not dominate. Ground truth comes from a multi-source golden set—human reviewers, follow-up commits, static analysis tools, and frontier LLMs—with duplicate findings merged and every finding validated under a published rubric by an LLM judge. Senior engineers who had not built the dataset re-labeled every ground-truth finding and agreed with ReviewBench 96.6% of the time.

The benchmark reports six metrics in two families: grounded precision, recall, and F1 against the fixed golden set, and augmented precision, recall, and F1 that give agents credit for valid issues not previously known. That matters because a fixed golden set inevitably becomes incomplete as systems get more capable, and augmented metrics avoid penalizing agents for finding unanticipated problems. Users can slice results by severity and category and adjust a β parameter to favor broader coverage or lower noise, with the leaderboard re-ranking accordingly.

GitHub used ReviewBench to evaluate Copilot code review across iterations, and one recent experiment—a multi-model ensemble that combines several independent model runs into a single review—illustrated the benchmark's value. ReviewBench predicted higher precision, recall, and comment volume with lower cost per review; an online A/B test later showed addressed rate rose 8.0%, recall rose 13.6%, comment volume rose 61%, and cost per review fell 8.0%. Severity-level evaluation also matched, predicting a 227% increase in critical comments versus 262% online. The open question is whether this predictive accuracy will hold as review agents evolve and the fixed golden set ages; the test is whether future submissions still see offline signals track production.