AI-written code breaks production half as often as human code

Mergify analyzed every pull request merged through its queue over a rolling 90-day window, capturing more than 200,000 merges across 477 engineering teams. The goal was straightforward: to measure what actually happens to a pull request between approval and landing on main—the gap where a green PR can still break the branch due to conflicts with other queued changes, and where merges quietly pile up. Almost nobody measures this stretch, the authors note, because almost nobody has the data.

The most striking finding was that AI-assisted PRs broke main about half as often as those written without AI help. Of 153,000 merges analyzed (the filtered cohort running a real queue with at least ten merges and two human authors), AI-assisted PRs broke main at 1.9%, compared to 4.4% for non-AI code. The authors tested this result to rule out confounds. When they compared AI and non-AI PRs within the same repositories, AI came out lower in 21 of 27 repos where rates differed. When they sliced by PR size, the gap held in every bucket: AI stayed near 1.7% whether the change was ten lines or a thousand, while human PRs climbed from 3.2% to 5.3% as they grew. The intuition that AI PRs are simply smaller and therefore safer turned out backwards—AI-assisted PRs were larger on average, 137 changed lines versus 84. The authors are cautious about causation (teams that keep the AI footer visible may be more disciplined in ways the data cannot measure, and a PR in a failed batch may not have caused the break), but they conclude that "AI assistance is associated with fewer broken-main batches" and that the data does not support the concern that AI is making main more fragile.

The second key finding concerns the relationship between team size and broken-main risk. The rate scales steeply: 0.8% for teams of 2–5 engineers (roughly one break per 130 merges), 0.98% for 6–15 engineers, 2.49% for 16–40 engineers, and 12.5% for 40+ engineers (one break per eight merges). Queue time also scales: sub-minute for teams under 15 engineers, roughly 11 minutes for 16–40, and roughly 18 minutes for 40+. The curve bends noticeably between the 6–15 and 16–40 buckets, which is exactly where teams usually start feeling merge pain. At small scale, a broken main is rare enough to handle by hand. At forty-plus engineers, it becomes "most of a team's week." The mechanism is straightforward: more engineers means more PRs in flight at once, and more chances that two independent, individually-correct changes touch the same assumption. Failed batches on private repos bundle about six PRs on average, so one bad change can stall five good ones queued behind it.

Open-source and private codebases tell different stories. Open-source merges break main at 1.1% (one in 88), while private ones do so at 5.1% (one in 20). The authors attribute this to the structure of work rather than code quality. Open-source contributions are often isolated—a fix or feature from someone touching one corner of the codebase. Private monorepos concentrate the opposite: dozens of engineers editing interlocking code on a shared deadline, which creates far more opportunities for two independent changes to collide. Failed batches on private repos bundle roughly 5.8 PRs on average versus 2.6 in open source. AI assistance also shows up differently: it appears in 14.4% of private merges but only 7.6% of open-source ones. This gap reflects tool adoption patterns, not code safety.

The analysis also finds that most teams are not using merge queues effectively. Batching tests several PRs together in one CI run and merges them as a group if they pass—the main lever for cutting CI spend on a busy queue. Yet 94% of private merges still go through one PR at a time. Only about 6% ride in a batch (2.7% in open source), and when they do the batch holds four PRs on average, sixteen at most. The authors note that teams are "leaving that saving on the table, probably because the default feels safe and nobody has gone back to tune it." More broadly, the data shows that PR time in queue is not a bottleneck: the median is about seven minutes, and 90% of PRs clear within the hour, even on large teams. Throughput averages around three merges per active day for a typical repo, with the busiest ten percent handling thirteen or more. Automation like Dependabot and Renovate clears the queue in about 0.4 minutes, against roughly twelve for a human PR, flattening the human numbers somewhat.

AI assistance already appears in about one in seven private merges—marked by a "Generated with" footer or co-authored-by line. This is described as a floor, not a ceiling, since most tools that leave no trace (inline Copilot, Cursor) are invisible to the measurement. Autonomous agents authoring whole PRs are still a rounding error, a few hundred merges out of 153,000. What the data captures is simpler: most of a team writing code with a model open in the next pane, then shipping it through the same queue as always.

The analysis draws on anonymized data from 477 engineering teams using Mergify's merge queue service, covering a rolling 90-day window. The core insight—that AI-assisted code breaks main half as often—emerged despite the authors' initial caution. They tested the finding three ways: by comparing AI and non-AI PRs within the same repositories (AI came out lower in 21 of 27 repos where rates differed), by slicing results across PR sizes (the gap held in every bucket), and by accounting for code volume (AI PRs were actually larger, 137 lines versus 84). The finding is framed as observational rather than causal—the teams that keep the AI footer visible may be more disciplined in unmeasured ways—but both easy alternative explanations (smaller, simpler AI code, or luck) do not survive scrutiny.

A secondary but arguably more consequential finding concerns team size. The broken-main rate is not uniform; it scales about 16-fold from small teams (0.8% at 2–5 engineers) to large ones (12.5% at 40+). This threshold matters because merge queues are the primary tool that prevents a green PR from breaking main when multiple independent changes collide. Below fifteen or so engineers, a queue is optional and most teams handle failures by hand. Above that, the standing tax of broken-main events becomes substantial enough that a queue earns its place. The gap between 6–15 and 16–40 engineers is where the curve bends noticeably, aligning with where most growing teams report first feeling merge pain.

A third notable gap exists between open-source and private codebases. Open-source repos break main at 1.1%, while private ones do so at 5.1%—nearly a five-fold difference. The authors attribute this not to code quality but to the structure of work: open-source contributions tend to be isolated (a single contributor fixing one corner), whereas private monorepos concentrate interdependent work (dozens of engineers editing interlocking code on a shared deadline). Failed batches on private repos bundle roughly 5.8 PRs on average versus 2.6 in open source, meaning a single bad change stalls more work behind it. This suggests the risk is fundamentally about the collision of assumptions across a codebase, not the source or authorship of the code itself.