BigSleep (45) edges Fuzzilli (37) on V8 bugs in Google test

Project Zero's comparison rests on two systems that both run in production against newly added V8 code: Fuzzilli, a generative JavaScript engine fuzzer that starts picking up changes within hours, and BigSleep, an agentic LLM system that runs once per day and is prompted to review new commits and write proof-of-concept testcases. Both report through Clusterfuzz and file into Chrome's issue tracker, which makes the bug counts fairly directly comparable. The raw production numbers required several rounds of filtering — removing non-security bugs and unactionable reports — before settling at 45 versus 37 actionable vulnerabilities.

The overlap question was approached two ways. In the bug tracker, Project Zero traced duplicates across filings, Clusterfuzz's underlying database, and shared fix commits to arrive at roughly 25% overlap in production. In a dedicated experiment, it fed BigSleep the commits that likely introduced 112 historic Fuzzilli bugs and let the agent attempt to re-discover them; BigSleep re-found about 62% of the original bugs and found a different, by-then-fixed bug for another 12%. Project Zero says several factors probably suppress production overlap — once either system's bug is fixed the other can no longer rediscover it, and BigSleep's pipeline saw model upgrades and occasional automation hiccups since launching in April.

On difficulty, the data pointed against the idea that AI would simply pick up what fuzzers struggle with: bugs that took Fuzzilli longer to find were also likely to be hard for BigSleep, which Project Zero reads as a reason to keep fuzzers running. The per-component breakdown showed notably lower overlap in JIT compilers, the area where Fuzzilli specializes most. A memorization check across bug ages found no statistically significant effect before the model's effective knowledge cutoff, though Project Zero recommends re-evaluating that for future models.