LLMs have unfixable security flaw letting hackers bypass safeguards

The vulnerability uncovered by this research strikes at a core architectural choice in how modern LLMs work. Because models see all text as "one big sheet of tokens," they cannot naturally distinguish between a user's prompt, the model's own reasoning notes, system instructions, or external information the way humans can. To solve this problem, developers created a system of tags—<user>, <assistant>, <system>, <think>, <tool>—to mark different sources. However, the researchers discovered that models do not actually rely on these tags. Instead, they identify text roles by recognizing patterns in writing style and word choice. This means an attacker who can write text that looks like internal reasoning can fool the model into treating it as if it came from inside the model's own chain of thought, bypassing the safeguards the model is supposed to follow.

The implications are stark because this weakness is not a bug that training can fix—it is built into the fundamental way LLMs process and interpret text. Red-teaming efforts, where human testers and AI systems search for new vulnerabilities, can find and patch specific attacks, but the underlying mechanism remains. As Jasmine Cui notes, the approach of training models to resist attacks is "like watching The Simpsons and they have Bart writing 'I will not say something inappropriate to his teacher' a hundred times" and still failing. No list of forbidden behaviors is exhaustive, and attackers, as Cui's own experience shows, are endlessly creative—she has tricked models by making them impersonate a drunk person, or by convincing them they were already being used by the military for purposes they initially refused.

The practical consequence is that as LLMs move into critical roles—controlling military systems, managing health care, processing financial transactions—the security assumption underlying their deployment is flawed. Charles Ye's warning that "there's been no study of the fundamental science here. We're all doing it ad hoc" captures the problem: the technology is being deployed at scale before its core vulnerabilities are understood or solved.