What was the actual bug the accurate review found?

The keyword half of hybrid retrieval scores terms by how rare they are, but the code computed rarity over only the few dozen candidates already fetched instead of over the whole corpus. At a pool of 60, a term appearing in all 60 candidates scores near zero while a term appearing in one scores 3.7—the exact inversion of intent.

Why did the true bug fix not ship?

The pool-local rarity bug is real, but the fusion score only orders the wide candidate pool; a cross-encoder reranker owns the final top-5, so the pool-local inversion has almost no effect on the final pipeline. In A/B testing on the live corpus, the baseline found the right source for 76 of 83 questions and both versions of the fix found 75, so zero improvement was measured.

What false claims did the warmest review make?

It fabricated a cache not in the code, retry logic not in the code, a database failover path not in the code, and CI scanners plus nightly backups asserted sight-unseen. It also reported a missing rate limit that answers with a 429 in production, recited numbers from the author's public portfolio as 'verified' findings, and rated the buggy function best-in-class.

Large Language ModelsAI Coding AssistantsAI Safety & AlignmentHacker NewsAug 13, 2026Aug 14, 2026, 04:01 JST4 min read

Three AI models reviewed same code; top scorer praised the bug

Send on LINE

Key takeaway

Three frontier language models reviewed the same code bundle, and their performance ranged from perfect verification to widespread fabrication. The most reliable review (5 for 5 verified claims) found the only real bug—a hybrid retrieval scoring algorithm that inverted term rarity by computing statistics over a small candidate pool instead of the whole corpus.
When the fix was implemented and A/B tested on live data, it did not improve the system's accuracy (baseline 76 of 83 correct; both fixed versions 75 of 83), so despite being a true finding, it was reverted.
The warmest review, by contrast, fabricated several features and gave the buggy function the highest rating.

3 Key Points

What happened
A software platform's code was sent to three frontier LLM models for review. One review achieved 5/5 accuracy and identified a real algorithmic bug (pool-local rarity calculation inverted term scoring). A second flagged the buggy function but for the wrong reason. The third gave a near-perfect score but fabricated features not in the code, including praising the buggy function as best-in-class.
Why it matters
The accurate finding—correcting hybrid retrieval to score term rarity over the whole corpus instead of just fetched candidates—was implemented and A/B tested against live data. The baseline answered 76 of 83 questions correctly; both versions answered 75, so the true bug fix did not ship. This shows that even verified findings can fail real-world measurement, and that LLM code reviews require claim-by-claim verification against source code, not trust in the review's confidence level.
What to watch
The falsest review (the 'glowing one') fabricated a cache, retry logic, and database failover that do not exist in the code; claimed 'zero data egress' when answers actually ship retrieved text to a cloud model; and recited the author's public portfolio back as 'verified' findings. The author pinned a regression test for the rejected bug fix so that if the reranker's role changes, the finding can be re-measured without re-debate.

Ask the AI about this article →

Summaries like this, in your inbox every morning.

Get news like this every morning — free

Context & Analysis

The three reviews demonstrate a hierarchy of reliability in LLM code review that does not correlate with confidence or warmth of tone. The accurate review asserted nothing the bundle could not support, yielding five findings all verified against source—including the only real algorithmic bug any reviewer caught. The second review flagged the correct function but attributed the defect to a per-query cost problem that does not exist, and also hedged its claims transparently where the bundle did not show evidence. The third review, despite earning the highest confidence score, fabricated the most: caches, retry paths, failovers, and data-egress claims that did not survive contact with the code. Notably, it also praised the exact function containing the bug as best-in-class, demonstrating that high confidence in false positives can be more dangerous than modest accuracy, because the subject is tempted to quote the warmest praise.

The second critical lesson emerges from the fate of the true finding. The pool-local rarity bug is real and was correctly identified and implemented; the fix computed term rarity statistics over the whole corpus as intended. Yet when A/B tested against the live corpus, both the baseline and the fixed version answered 75 of 83 questions correctly (down one from baseline's 76), so the fix was reverted. This outcome reveals that architectural constraints—here, the reranker's dominance over the final ranking—can render a verified bug fix ineffectual in practice. The author pinned a regression test so that future changes to the reranker's role would re-measure rather than re-debate the fix, turning rejection into a conditional condition rather than a permanent verdict.

The work also uncovers a methodological flaw: when a reviewer points at the right target for the wrong reason, the natural response is to refute the reason and move on. Instead, audit the target afresh for the real defects hiding in the lines just read. Verification must also extend to praise, not only findings, because false positives are the quotable half of a review and can mislead users into shipping dangerous recommendations—such as moving secrets from host-external injection into a file inside the repository tree, dressed as a best practice.

FAQ

What was the actual bug the accurate review found?: The keyword half of hybrid retrieval scores terms by how rare they are, but the code computed rarity over only the few dozen candidates already fetched instead of over the whole corpus. At a pool of 60, a term appearing in all 60 candidates scores near zero while a term appearing in one scores 3.7—the exact inversion of intent.
Why did the true bug fix not ship?: The pool-local rarity bug is real, but the fusion score only orders the wide candidate pool; a cross-encoder reranker owns the final top-5, so the pool-local inversion has almost no effect on the final pipeline. In A/B testing on the live corpus, the baseline found the right source for 76 of 83 questions and both versions of the fix found 75, so zero improvement was measured.
What false claims did the warmest review make?: It fabricated a cache not in the code, retry logic not in the code, a database failover path not in the code, and CI scanners plus nightly backups asserted sight-unseen. It also reported a missing rate limit that answers with a 429 in production, recited numbers from the author's public portfolio as 'verified' findings, and rated the buggy function best-in-class.

Hacker NewsRead Original Article

Send on LINE

Get the latest Large Language Models news every morning

For example, today's edition would include:

DataAgent launches with $10M to auto-fix Kubernetes faultsSiliconANGLE AI · 43m ago
SK Hynix custom HBM boosts inference up to 5.15xDIGITIMES Asia · 43m ago
Nvidia Earnings: Boring by Design, Avoiding a Consolidated WorldStratechery (Ben Thompson) · 43m ago

AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.

Subscribe free →

Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →