How are Catching JiTTests different from LLM-based code review?

Instead of reviewing code the way a human reviewer would, Catching JiTTests generate executable tests that demonstrate a concrete behavioral difference introduced by the change. This requires more computational steps but catches classes of bugs that LLM- or agent-based code reviews might not.

What is the human-on-the-loop approach?

Rather than reviewing every output, engineers set criteria and review outcomes at an aggregate level, confirming whether an observed behavioral change was intended. The goal is to maintain consistency of human labeling while reducing the likelihood of an engineer being brought in to review a weak catch.

How does the system filter false positives?

It combines a traditional rule-based approach called RubFake with an ensemble called LLM-as-Judge, which aggregates judgments from several state-of-the-art LLMs to classify weak catches as likely true or false positives. Working together, they produce a ranking of candidates worthy of further consideration.

Large Language ModelsAI Coding AssistantsAI Safety & AlignmentMeta Engineering1h agoOct 30, 2026, 01:00 JST

Meta: Catching JiTTests caught 74 serious privacy bugs before code landed

Send on LINE

3 Key Points

What happened
Meta deployed Catching JiTTests — LLM-generated tests designed to fail on buggy code — and says they caught 74 serious (SEV) privacy bugs at the diff stage in the first half of 2026, with a 60% true-positive rate.
Why it matters
The results indicate AI-generated tests can serve as a safety layer that catches bug classes traditional tests and LLM-based code review may miss, with human engineers only confirming whether a behavior change was intended rather than reviewing every output.

WHO IT HITSEngineering teams at large codebases — particularly those adopting agentic software development that speeds up code shipping — may see AI-generated tests as a way to catch bugs before production without adding per-output review load, though whether the true-positive rate holds outside Meta's environment remains to be seen.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Get these free every morning

Context & Analysis

Meta's disclosure caps roughly a year of public research updates on applying large language models to software testing. The company first unveiled the JiTTests approach, then spent five months deploying Catching JiTTests to test scalability and applicability beyond a research setting. The system now runs as part of Meta's infrastructure rather than as a prototype.

The deployment addresses a problem specific to Meta's scale: hundreds of millions of lines of code are constantly evolving, and traditional testing struggles to keep pace. Catching JiTTests approach that problem from two directions at once — an intent-aware workflow that infers what a code change is trying to do and targets likely failure points, and a dodgy diff workflow that treats every change as a potential bug and generates tests to expose behavioral differences. False positives are filtered through RubFake and the LLM-as-Judge ensemble before engineers are asked to weigh in.

The reported outcome is that 589 engineers engaged with the system, and those who did reported it surfaces genuinely actionable findings at a 60% true-positive rate. Meta also reports that Catching JiTTests helped lower the overall occurrence of privacy SEVs by 20% during the first half of 2026, framing the system as a scalable safety layer for AI-generated testing rather than a replacement for human judgment.

FAQ

How are Catching JiTTests different from LLM-based code review?: Instead of reviewing code the way a human reviewer would, Catching JiTTests generate executable tests that demonstrate a concrete behavioral difference introduced by the change. This requires more computational steps but catches classes of bugs that LLM- or agent-based code reviews might not.
What is the human-on-the-loop approach?: Rather than reviewing every output, engineers set criteria and review outcomes at an aggregate level, confirming whether an observed behavioral change was intended. The goal is to maintain consistency of human labeling while reducing the likelihood of an engineer being brought in to review a weak catch.
How does the system filter false positives?: It combines a traditional rule-based approach called RubFake with an ensemble called LLM-as-Judge, which aggregates judgments from several state-of-the-art LLMs to classify weak catches as likely true or false positives. Working together, they produce a ranking of candidates worthy of further consideration.

Meta EngineeringRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.