Chain-of-Thought Spoofing Reveals Unfixable Flaw in LLMs

The paper presented at ICML in July challenges the current industry approach to AI safety. Companies have relied on red-teaming—hiring human security testers to find new attack methods—and increasingly on automated "LLM hackers" like OpenAI's GPT-Red to identify model weaknesses. The goal has been to patch these specific vulnerabilities through retraining. However, this strategy assumes threats can be comprehensively enumerated and defended against in advance.

The research exposes why that assumption fails. LLMs are fundamentally vulnerable because of how they process and respond to input. The Chain of Thought mechanism, designed to help models reason through problems by generating internal notes, can be spoofed. Once an attacker understands the style and structure of a model's reasoning, they can inject fake reasoning that the model accepts as its own. This is not a question of bad training data or weak guardrails—it is a structural property of how these systems work. Researchers Qui and Ye use an apt analogy: telling an LLM "do not do X" is like the Simpsons scene where Bart is forced to write "I will not say inappropriate things" 100 times, yet continues to misbehave anyway. Every "do not" list is incomplete by definition, and an attacker needs only one exploit to succeed.