GitHub opens ReviewBench, a 219-PR code review benchmark

Existing benchmarks for AI code reviewers tend to trade off label quality, coverage and real-world representativeness, which leaves teams without a reproducible way to compare systems. ReviewBench is GitHub's attempt to close that gap by modeling its corpus on over 100 million real pull requests on GitHub and weighting pull-request size toward the reviewable middle and tail rather than tiny single-file changes.

Its ground truth is assembled from human reviewers, issues inferred from author follow-up commits, deterministic analysis tools and multiple frontier LLMs, then deduplicated and judged under one published rubric. Reviewer agents are scored on two metric families: grounded precision and recall against existing gold-set labels, and augmented precision and recall that can credit valid findings no gold-set producer surfaced. Results can be sliced by severity and category, or re-ranked by adjusting β in the Fβ score for teams that prefer broader coverage or less noise.

GitHub says it has used the benchmark to evaluate Copilot code review across successive iterations, and that offline changes have consistently pointed in the same direction as later production A/B tests. The stakes therefore hinge on whether that offline-to-online alignment holds for agents built outside GitHub, which the public leaderboard and self-serve runner are designed to test.