AI patches fix security flaws only 26% of the time without human review

What happened

Researchers at 1Password's Off-by-1 Labs tested security patches generated by ChatGPT 5.5 and Claude Opus 4.8 across six recently disclosed CVEs, producing 6,080 patches total. Only 26.0 percent fully resolved the vulnerability without changing application behavior; 20.1 percent fixed the issue but altered how the application works, 2.3 percent introduced new security issues, 49.3 percent failed to fix at least one existing exploit path, and 2.2 percent both failed to fix the vulnerability and opened a new one.

Why it matters

The authors conclude that the expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin. Even successful patches often prove fragile—more than a third in the clean or behavior-changing categories didn't address the underlying problem. While a single successful patch costs just $6.74 on average, human review overhead may exceed the cost of engineers patching vulnerabilities themselves using standard LLM-assisted techniques.

What to watch

The success rate for LLM patches depends heavily on initial guidance—jumping to 65.0 percent with correct guidance but plummeting to 15.2 percent with incorrect guidance. The researchers have released a patch evaluation harness called FLAWED that organizations can use to test their own LLM-assisted security fixes.