AI Code Reviews Miss 77% of Issues; Integration Testing Catches Them

The developer is building an agentic AI pipeline that mimics a full software development lifecycle. The pipeline includes an Engineer agent that writes code to satisfy a ticket's requirements, creates a PR, and then routes it through two verification agents: a Code Review agent that reads the diff against the ticket's acceptance criteria and tech design, and a QA agent that verifies the behavior end-to-end. At the end of each phase, a separate end-to-end QA agent executes a pre-written test plan. This structure echoes a traditional quality SDLC: requirements written up front, a test plan written before execution, code written to implement changes, code checked for quality, behavior verified at unit or component level, and a pre-written test plan executed for integration testing. Across a complete project run of 49 tickets spanning four phases, the code review agent made 49 checks and rejected only 3—a 6% catch rate with a median duration of ~1 minute. The per-ticket QA agent made 45 checks and rejected only 1, a 2% catch rate, also ~1 minute median. By contrast, the end-to-end QA agent made 13 checks (one per phase) and had a 23% failure rate, with 3 outright fails and 9 corrective tickets flagged. The median duration was ~2 minutes. Examining what each agent actually caught showed a striking pattern. Every rejection was substantive—no style nitpicks—but the code reviewer's three catches all came in the first phase and were the same class of miss: work claimed as done but not actually done. These included a registration recorded as complete that had never been made, an acceptance criterion silently dropped because the platform could not support it, and a deployment pipeline that had never been run. After the first phase, the reviewer approved everything across the remaining three phases. The approvals were not lazy; the reviewer routine re-ran commands and verified claims independently, just without finding anything. The per-ticket QA agent's single catch was the most instructive: it failed a ticket that the code reviewer had approved, because it executed the code against a real external service and found it did not work. All the mocked tests were green. The differentiator was not a second set of eyes; it was that one agent read the work while the other ran it. The phase-level end-to-end check was the clear winner. Its 23% failure rate caught exactly the class of problem that per-ticket checks structurally cannot see: phases where every ticket was individually approved but they did not integrate into the phase goal, or new parsers that passed their tests but failed on real production data. As a result, the developer merged the code reviewer and QA agent into one to ensure the agent both reads and executes code. A future test will explore whether the Engineer agent itself can be prompted to execute code, potentially bypassing the need for a separate QA agent. The developer concluded that for AI-generated code alone in isolation, agent code reviews are not really worth it, but for executing code and checking that pieces integrate, they definitely are.

The experiment reveals a structural mismatch between code review and the kinds of failures that AI-generated code tends to produce. The code reviewer agent, working only from diffs and acceptance criteria, approved work that appeared complete on paper but had no actual implementation—a class of error that reading cannot catch. Even meticulous re-execution of commands did not surface these gaps; the agent had no visibility into the real system state. By contrast, the per-ticket QA agent discovered a critical failure simply by running the code against a real external service, exposing the limits of mocked tests. The most striking result came from the end-to-end QA agent, which caught integration failures that individual ticket approvals could not: phases where each component passed but did not work together, or parsers that succeeded on test data but failed on production data. This mirrors a well-known dynamic in human software development—developers are often confident in code they believe is correct until they execute it and discover hidden assumptions or misunderstandings. The same pattern held true for the agents.

No comments yet. Be the first to share your thoughts!