AIToday
Large Language ModelsAI Coding AssistantsZenn AI/MLPublished: Nov 17, 2026, 04:00 JST

FSTab maps AI coding habits to hidden flaws, hitting 85.67% match

Send on LINE
FSTab maps AI coding habits to hidden flaws, hitting 85.67% match

3 Key Points

  1. What happened

    The FSTab study analyzed 1,050 generated programs from six coding models using CodeQL and Semgrep, then predicted likely vulnerability classes for unseen apps from their visible features. For Claude-4.5 Opus with Semgrep in a cross-domain test, 85.67% of apps had at least one predicted rule match a detection.

  2. Why it matters

    The 85.67% figure measures agreement with static-analysis detections, not a break-in success rate. It suggests source-blind reviewers could use a model's track record to set testing priorities, though the authors note it is not a model safety ranking.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

FSTab builds its tables in a white-box stage: researchers generate many apps from the same model while varying prompts and domains, scan the code with CodeQL and Semgrep, and classify detections by the user action involved. The paper standardizes user actions into 59 types and uses context such as function names, identifiers, routes and API calls, so apps that look different can be compared at the level of "logging in" or "sending a file." For unseen apps, the researchers observe public UI, DOM and network behavior, map what they find to the same categories, and look up the table for the source model.

The evaluation varies prompt wording and app domain to test whether weak spots persist, including a cross-domain condition that excludes the target domain when building the table. The paper is careful about what the numbers mean: an appendix discusses a manual audit of detections, and individual examples in another appendix do not convert the main results into intrusion rates. That framing places FSTab alongside, not instead of, conventional static application security testing, which examines a specific repository's implementation; FSTab aggregates detections across many generated products to find repeated failures.

FAQ
Does the 85.67% figure mean attackers can break into that share of apps?
No. The paper defines it as the share of apps where at least one predicted vulnerability rule matched a static-analysis detection. It is not an intrusion success rate and should not be used as a model safety ranking.
How many programs and models were evaluated?
The study evaluated 1,050 generated programs from six code-generation models: 900 from WebGenBench and 150 from E2EDev.
Does FSTab work without access to an app's source code?
The prediction step for unseen apps is treated as black-box: it observes public UI, DOM and network behavior to classify features, then looks up the source model's table. Only the table-building stage reads its own generated source.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleTOKIUM to list on TSE Growth Market on 25th