What is prompt injection?

It is a new class of security weakness where an attacker sneaks an instruction into an LLM's input, hoping the model will obey it, such as telling it to send over social security numbers it knows.

What is capability-based security?

It gives each program only the authority it needs for the task at hand, rather than everything its user is allowed to do, and was built in part to solve the "confused deputy" problem where a program is tricked into using its legitimate power for someone who has none.

Who was Peter Eckersley?

He was a co-worker of the author at EFF who also helped create Let's Encrypt and Privacy Badger, and he passed away in 2022.

AI Safety & AlignmentLobsters AI2h agoNov 13, 2026, 01:00 JST

Black October: AI-found flaws show prompt injection is social engineering

Send on LINE

3 Key Points

What happened
The author, writing on day 42 of what he calls the "Black October" wave of AI-discovered security flaws, says Peter Eckersley's prediction that near-superintelligent AIs would find and exploit flaws ecosystem-wide has come true.
Why it matters
Prompt injection, which the author calls "social engineering silicon", exploits the same human-style tricks, and he argues capability-based security, giving each program only the authority it needs, can narrow the blast radius but will not be sufficient.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Get these free every morning

Context & Analysis

The author traces the current wave of AI-discovered flaws back to conversations at EFF in the 2010s, when Peter Eckersley warned that near-superintelligent AIs would soon be used to find and exploit security flaws across the whole ecosystem. Eckersley, who helped create Let's Encrypt and Privacy Badger, died in 2022 before he could tackle the issues he had foreseen.

Alongside pre-existing flaws, the author argues modern transformers have introduced prompt injection as a new class of weakness, and he draws a direct parallel to social engineering: attackers exploit empathy, urgency, or assumed authority in humans, and now use similar tricks on models. He describes capability-based security as the fix security experts pointed to, noting that several projects have since matured, including Microsoft, ARM, SRI and Cambridge University's CHERI at the CPU level, Google's Fuchsia, FreeBSD's Capsicum, Cloudflare's Workers, and the Spritely Institute's OCapN networking layer.

The author says he left EFF in 2021 to push this work and secured funding from the Filecoin Foundation for the Decentralized Web for the Spritely Institute. Even so, he concludes that capability-based security narrows the blast radius of social engineering but is not sufficient on its own.

FAQ

What is prompt injection?: It is a new class of security weakness where an attacker sneaks an instruction into an LLM's input, hoping the model will obey it, such as telling it to send over social security numbers it knows.
What is capability-based security?: It gives each program only the authority it needs for the task at hand, rather than everything its user is allowed to do, and was built in part to solve the "confused deputy" problem where a program is tricked into using its legitimate power for someone who has none.
Who was Peter Eckersley?: He was a co-worker of the author at EFF who also helped create Let's Encrypt and Privacy Badger, and he passed away in 2022.

Lobsters AIRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.