Google Mantis-style scan hits 0 false positives in 30-question test

The benchmark is framed around a practical complaint: LLM code review tends to produce a flood of warnings that turn out to be harmless, and developers stop reading them. Nishi-tō cites academic work putting the true-positive rate of simple prompt-based AI vulnerability scanning below 7%, which is the baseline this test tries to beat.

The four approaches were chosen to represent generations of tooling. Bandit, a Python AST pattern matcher, caught only 5 of 15 real flaws and raised 4 false alarms, because it cannot see language quirks like os.path.join discarding its first argument when the second is an absolute path. Semgrep improved recall to 60.0% but still raised 6 false positives, failing to track validation that lives in a separate file or an authorization decorator.

The raw LLM result is the pivot of the piece. It caught 14 of 15 real flaws, including the os.path.join trap, double decoding and Zip Slip, but consumed about 7,700 tokens and over 40 seconds and still flagged 3 safe snippets. The Mantis-style design answers both problems at once: an AST summary tree cut input tokens from 7,735 to 1,543 (roughly 80%), and a sandboxed dynamic proof-of-concept step rejected every safe decoy, leaving precision at 100.0% while recall held at 93.3%. The author notes that production use would still require container-escape defenses and autonomous mock generation for external dependencies, which are operational concerns rather than benchmark scores.