DeepSeek V4 Pro tops 10 AI models in cybersecurity benchmark at fraction of frontier cost

The benchmark represents a significant shift in the publicly available AI model hierarchy for security tasks. Prior evaluation showed open-weight models catching up; this benchmark demonstrates they have now caught up and in some cases surpassed frontier models on the specific task of vulnerability discovery. The key insight is that vulnerability discovery—unlike many AI tasks—benefits from output variance: running the same model multiple times on the same problem yields different exploration paths, and pooling those results recovers more vulnerabilities than any single run. DeepSeek V4 Pro found only 17 vulnerabilities in its first pass but reached 28 across three runs; Qwen climbed from 19 to 26. This property flips the usual penalty of model inconsistency into an advantage.

Cost efficiency emerges as a critical new variable. Three runs of DeepSeek Flash cost $108 and recovered 24 vulnerabilities—the same as Grok's best single run at roughly 10 times the price. GLM-5.3, tested as an early evaluation partner, improved from preview to final release, matching Opus 5's recall at 69.8% lower cost. However, the trade-off is noise: open-weight models produced more false leads, which the researchers explicitly call out as a downstream burden. This means cost savings shift the human effort from running the model to filtering its output, a detail that complicates the simple "cheaper is better" narrative.