Code Review Cannot Fix AI Coding Assistants' Core Problem, Research Shows

The author, a skeptic of AI coding assistants in software development, frames the central problem not as intellectual property concerns or resource consumption, but as a practical issue: there is no scientific evidence that AI coding assistants help developers write code better or faster. The author's frustration centers on the fact that proponents of these tools rarely engage with this evidence when defending their use.

The most common counterargument from defenders is that AI-generated code should be treated like code from junior developers or interns: reviewed thoroughly before being committed. The logic is straightforward—developers already review code before merging it, so reviewing AI output should be no different. However, the author points out that this defense ignores empirical research on code review practices. Scientific studies have established two critical constraints: code reviews lose effectiveness after 1 hour of continuous attention, and reviews conducted faster than approximately 400 lines of code per hour fail to effectively catch defects. These limits are not arbitrary preferences but are rooted in how human attention and fatigue work.

When these empirical findings are applied to AI coding assistants, the math becomes grim. For every 400 lines of code an AI assistant generates, a senior developer must spend 1 hour reviewing it. A developer has between 10 to 40 such focused review slots per week, and these are often interrupted by meetings, urgent work, and other demands. In the best case—where most code is simple boilerplate—a developer using an AI assistant might commit a few thousand lines of code per day. More realistically, it is fewer than 1,000 lines per day, including boilerplate, tests, migrations, and configuration. This is barely faster than writing the code by hand.

But the problem deepens. Recent research suggests that humans reviewing AI-generated code are less effective at catching defects than when reviewing human-written code, yet they feel more confident that they have found all errors. This creates a dangerous situation: the review process does not actually solve the problem it purports to address. The output of a human reviewer paired with an AI assistant is lower quality than a human reviewer paired with a human coder, yet the reviewer incorrectly believes they have done a thorough job.

The author uses a striking example to illustrate the risk: one prominent AI advocate suggested that AI assistants should be used to write shell scripts and bash code—exactly the kind of code that is hardest to review and most prone to catastrophic failures if a single character is wrong. Recommending that the code hardest to review correctly be generated by a system that frequently produces errors seems, to the author, to represent a failure to engage seriously with the review question at all.

The author concludes by acknowledging the possibility of being wrong and calls for the kind of empirical evidence that could prove the critique mistaken: peer-reviewed research on how effectively humans can review AI-generated code, how fast they can do it, and how many defects they catch. Until such data exists, the claim that code review solves AI coding assistants' reliability problem remains unproven.

The article presents a fundamental critique of the most common defense of AI coding assistants: that human code review mitigates their tendency to generate incorrect code. The author grounds this critique in empirical research on code review practices, which establishes two hard limits: reviews become ineffective after 1 hour of continuous concentration, and reviews above 400 lines of code per hour fail to catch defects effectively. These are not arbitrary preferences but documented findings from scientific study of human reviewers examining human-written code.

When applied to AI assistants, these limits create a severe practical constraint. If a developer uses an AI assistant to write code, they must then spend approximately 1 hour reviewing every 400 lines generated—a process that occupies all or most of their available deep-work time slots per week. In practice, this caps productivity at fewer than 1,000 lines of production-ready code per day, which is minimally better than what a developer would produce without assistance and may offer no real speedup once the review overhead is factored in.

The situation is made worse by a critical empirical finding: humans reviewing code generated by AI systems find fewer actual defects than when reviewing human-written code, yet report higher confidence in their reviews. This means the standard defense—"just review everything thoroughly"—does not merely impose a productivity ceiling; it may actually fail to catch AI-generated errors while creating false assurance. The author therefore concludes that until empirical research specifically demonstrates that humans can review AI-generated code as effectively as human-written code, the "review it all" argument is not credible.

No comments yet. Be the first to share your thoughts!