
What happened
The FSTab study analyzed 1,050 generated programs from six coding models using CodeQL and Semgrep, then predicted likely vulnerability classes for unseen apps from their visible features. For Claude-4.5 Opus with Semgrep in a cross-domain test, 85.67% of apps had at least one predicted rule match a detection.
Why it matters
The 85.67% figure measures agreement with static-analysis detections, not a break-in success rate. It suggests source-blind reviewers could use a model's track record to set testing priorities, though the authors note it is not a model safety ranking.
Summaries like this, every morning.
FSTab builds its tables in a white-box stage: researchers generate many apps from the same model while varying prompts and domains, scan the code with CodeQL and Semgrep, and classify detections by the user action involved. The paper standardizes user actions into 59 types and uses context such as function names, identifiers, routes and API calls, so apps that look different can be compared at the level of "logging in" or "sending a file." For unseen apps, the researchers observe public UI, DOM and network behavior, map what they find to the same categories, and look up the table for the source model.
The evaluation varies prompt wording and app domain to test whether weak spots persist, including a cross-domain condition that excludes the target domain when building the table. The paper is careful about what the numbers mean: an appendix discusses a manual audit of detections, and individual examples in another appendix do not convert the main results into intrusion rates. That framing places FSTab alongside, not instead of, conventional static application security testing, which examines a specific repository's implementation; FSTab aggregates detections across many generated products to find repeated failures.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
Fierce Network reports that AI agents are operating in production environments, with the implication that they…

Uber Engineering detailed a framework with three layers — provisioning through its Uber AI Registry, with sour…

CrowdStrike was named the Visionary Leader in the Frost Radar™: AI Security Operations Centers, 2026, position…

ServiceNow selected Copilot Studio and launched a Microsoft AI Agent Creation Portal, where its makers can bui…

Verizon CEO Dan Schulman said AI in customer service delivers a 1,200-basis-point or more improvement in custo…

Gartner's Shikauchi described a three-layer physical AI stack (cloud, on-site device, edge) and a path from Hu…
