AIToday
Large Language ModelsAI Safety & AlignmentZenn AI/MLPublished: Oct 20, 2026, 10:00 JST

Mitate halts prompt injection by blocking changes, not detecting attacks

Mitate halts prompt injection by blocking changes, not detecting attacks

3 Key Points

  1. What happened

    R2-san's Mitate (a tool that lets non-technical staff build business apps by chatting) stops indirect prompt injection (hidden instructions inside documents the AI reads) by structure, not detection: its design AI has no function-calling, activation is classified outside the AI, and changes made after reading past records are held for human confirmation. In testing, all 5 attacks via past records hit the confirmation card.

  2. Why it matters

    Asking a model to ignore embedded instructions is only a request, R2-san notes, so injected content can still slip through if it falls under an "addition" that needs no approval; deciding outside the AI whether a change goes live appears to close that gap, while small everyday additions stay friction-free.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Context & Analysis

Mitate is a product for people who are not IT specialists: they describe a need in conversation, and the system builds a business app such as attendance collection or equipment lending, which they can then adjust while using it. That makes it a natural target for indirect prompt injection, because the AI is routinely asked to read outside material. R2-san's first instinct — telling the model in its system prompt that what it reads is data, not instructions — is described as a request, not an enforcement, and no method fully detects injection.

The three layered measures therefore move the decision away from the model. The design AI can output only a JSON definition, a query for past records, and a reply to the user, so SQL, shell commands, credentials and notification destinations have no place to be written. Checks on the definition happen on the execution side, and notification recipients come only from roles declared in the app definition. Activation is likewise decided by the execution platform, not the AI, which is why a request to skip confirmation still produced a confirmation card in testing.

The remaining hole was the addition category, which goes live without confirmation to keep small edits light. Because the embedded instructions in the test stayed within that category, R2-san added a rule keyed to what the calling program placed in the AI's input: if past conversation or past design records were included, even an addition goes to the card. R2-san is explicit about the boundaries — the AI itself reads the record text, the CaMeL-style split of reading and designing models is not used, and Mitate does not claim to prevent prompt injection, only to close the paths by which a disturbed result could take effect without a human in the loop.

FAQ
What is indirect prompt injection?
It is when instructions hidden inside material the AI is asked to read — such as meeting notes or past conversations — cause the AI to follow them. R2-san treats this as the central risk for features where reading a document changes something.
How does Mitate decide when to show a confirmation card?
Mitate classifies a proposed change as equivalent, addition, or change; only "change" requires confirmation. A later rule also routes additions made after reading past records to the card, and the calling program records what was fed to the AI rather than asking the AI itself.
Did any attack get through in the tests?
No. Across 11 valid trials of the embedded-instruction attack, 0 reflected the injected instructions, and after the new rule all 5 record-based attacks stopped at the confirmation card. R2-san cautions that the AI simply never complied, and that success rates cannot be claimed given the small number of tests.

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleScience editor's "Slower, worse" AI piece used ChatGPT, essay says