OpenAI's GPT-Red AI super-hacker boosts model defenses against cyberattacks

OpenAI has deployed an internal AI system called GPT-Red—an LLM trained to attack other AI models—as a tool to systematically strengthen the defenses of its most advanced models before release. The company released GPT-5.6, its latest flagship LLM, last week, and states that training it against GPT-Red made the model its most robust release yet.

GPT-Red automates a safety evaluation process called red-teaming, traditionally performed by human security testers who attempt to find and exploit system vulnerabilities. To build GPT-Red, OpenAI's researchers took an untrained LLM and placed it in a self-play training loop alongside several other models. Over many rounds, GPT-Red improved at attacking while the other models improved at defending. The training took place in a simulated environment OpenAI designed to mimic real-world deployment scenarios, including web browsing, email and calendar reading, and code editing. When GPT-Red discovered a new attack, it would explore multiple versions to find the most efficient one for specific scenarios. Hunn, a research scientist and co-creator, explains: "Compared to a human red-teamer, the model is very, very good at finding exactly what will work, exactly what's most effective. It's extremely persistent about drilling down into an attack that it has discovered."

The most significant discovery was a novel attack type called a fake chain of thought. A chain of thought is a mechanism in which an LLM maintains internal notes and tracks partial results as it solves problems. GPT-Red found a way to inject false entries into another model's chain of thought, tricking it into acting on spoofed information. Chris Choquette-Choo, another research scientist on the team, illustrated the concept: "It's like if I told you that 1+1=3 and that you have verified this already. The model's like, 'Oh, okay, of course,' and it just spits out 3." OpenAI focused most of its effort on a different class of attack called prompt injection, in which a hacker embeds hidden instructions in text to make an LLM perform unintended actions such as copying confidential data, sabotaging code, or generating harmful output.

To validate GPT-Red's effectiveness, OpenAI reran a 2025 experiment in which human red-teamers attempted to break an earlier version of GPT-5. When GPT-Red performed the same task, it was more successful than the humans at finding effective attacks. The company also tested GPT-Red against Vendy, a vending machine agent developed by Andon Labs, and GPT-Red successfully hacked it to change item prices and cancel orders. When OpenAI applied the strongest attacks GPT-Red had discovered to its models, more than 90% worked against GPT-5 (released in August last year) but fewer than 23% worked against GPT-5.6. Jessica Ji, a senior research analyst at Georgetown University's Center for Security and Emerging Technology, praised the self-play approach: "The results look very promising."

GPT-Red has notable limitations. It struggles with back-and-forth conversational attacks that human testers handle easily, and it is not yet strong at exploiting image-based attacks, which can pass text to models in prompt injection scenarios. OpenAI emphasizes that GPT-Red supplements rather than replaces human red-teamers—people can find attacks it misses and vice versa—and the company is exploring a hybrid approach in which it provides GPT-Red with human-discovered attacks and asks the model to find all variations. OpenAI will not release GPT-Red, confident that the intensive compute investment required to train such a system over more than a year would be prohibitively expensive for others to replicate.

OpenAI's development of GPT-Red reflects a fundamental challenge in AI safety: as language models become more capable and are deployed in increasingly complex real-world scenarios—such as agents that interact with code, websites, and external systems—the scope of potential attacks grows faster than traditional human-led testing can handle. The company's co-creators, Nikhil Kandpal and Dylan Hunn, frame this as a problem of both expanding risk surface and growing blast radius. Rather than scaling up human red-teaming teams indefinitely, OpenAI designed GPT-Red using a self-play training loop, a technique in which an attacker model and defender models iteratively improve against each other. This approach mirrors competitive training dynamics seen in other domains and allows the system to discover novel attack vectors that might not have occurred to human testers.

The concrete results support the value of the method. When GPT-5.6 was tested against the strongest attacks GPT-Red discovered, fewer than 23% succeeded—a dramatic drop from more than 90% success against the earlier GPT-5 released in August last year. This measurable improvement in robustness across versions demonstrates that the red-teaming process yielded actionable defense improvements. Notably, GPT-Red uncovered attack types researchers had not previously encountered, including the fake chain of thought injection, which exploits the model's reasoning process itself rather than only its input handling. The discovery of such novel attacks underscores the value of automated red-teaming at scale.

No comments yet. Be the first to share your thoughts!