OpenAI、AIで防御を鍛える攻撃モデル「GPT-Red」開発

OpenAI has created GPT-Red, a specialized language model designed to function as an adversarial sparring partner for testing the defenses of other large language models. The company released GPT-5.6, its latest flagship LLM, last week. According to OpenAI, GPT-5.6 is the most robust release to date because it was trained through interactions with GPT-Red.

GPT-Red automates a safety practice called red-teaming, which is traditionally carried out by human testers whose job is to find as many ways as possible to break or compromise a system. The discovered vulnerabilities are fixed before the final software is released. As LLMs become more complex and are increasingly deployed as AI agents that can interact with computer files, websites, third-party code, and other agents, it becomes difficult for human-only teams to anticipate every potential attack. According to Nikhil Kandpal, a research scientist and co-developer of GPT-Red at OpenAI, "The surface area for risk expands and the scope of impact broadens." The model was designed so that safety testing remains effective even as more capable models emerge: new attack methods can be discovered by a system that will continue working alongside future generations of models.

To build GPT-Red, OpenAI researchers embedded an untrained-as-attacker LLM into a self-play loop alongside multiple other models. GPT-Red played the attacker role while the other models played defense. Over many rounds, GPT-Red honed its attack skills and the other models strengthened their defenses. Training took place in what OpenAI describes as a "dojo"—an environment designed to replicate real-world scenarios in which LLMs operate, including web browsing, email and calendar use, and code editing.

OpenAI placed particular emphasis on "prompt injection" attacks, in which hackers secretly embed instructions into an LLM to make it perform unintended actions such as copying confidential information, sabotaging corporate codebases, or generating harmful or inappropriate output. Theoretically, such commands can be embedded in any text the LLM might read, from code to website text. When GPT-Red discovered a new attack, it explored variations to find which worked best in particular scenarios. According to Dylan Han, a research scientist and co-developer of GPT-Red, the model is "very good at determining what is effective and what is most effective" compared to human red teamers, and it "has the persistence to thoroughly dig into an attack once found." The research team has claimed that GPT-Red discovered previously unseen prompt injection attacks, including one they call "Fake Chain of Thought." This attack exploits the chain-of-thought mechanism—an internal record in which LLMs log intermediate results as they work through problems. GPT-Red found a way to write false records into another model's chain of thought, causing it to believe and act on false information. Chris Choquet-Tu, a research scientist on the team, explained it with an analogy: "It's like if I said, '1+1=3 and you already verified it.' The model says, 'Oh right, yes' and then answers 3.'"

Jessica Zee, a senior research analyst at Georgetown University's Center for Security and Emerging Technology (CSET) who researches AI security, evaluated OpenAI's self-play loop approach positively, calling the results "very promising." To assess GPT-Red's attacking power, OpenAI in 2025 recreated an experiment in which human red teamers had found vulnerabilities in the older GPT-5. When given the same task, GPT-Red succeeded in finding more effective attacks than the humans. OpenAI also tested GPT-Red against Vendy, an automated vending machine agent developed by Andon Labs, a company that provides real-world performance evaluation of agents. GPT-Red successfully hacked Vendy and managed to change product prices and cancel customer orders.

Regarding defense, OpenAI reported that when it tested GPT-Red's most powerful attacks against its own models, over 90 percent succeeded against GPT-5 (released in August 2024) but fewer than 23 percent succeeded against the new GPT-5.6. However, GPT-Red has limitations. It is not skilled at devising attacks that involve repeated dialogue cycles between attacker and target—something human attackers can do easily. It also remains insufficiently effective at prompt injection attacks using images to convey text to models. According to OpenAI, GPT-Red complements rather than replaces human red teamers: humans can find attacks GPT-Red misses, and vice versa. One current effort involves giving GPT-Red human-devised attacks and having it explore all variations. As Zee notes, "Human expertise will continue to be very important," and identifying which areas genuinely require human testing will be valuable. OpenAI does not plan to release GPT-Red publicly, viewing it as far more powerful than imitation models. The research team developed it over more than a year backed by substantial computational resources, and as research scientist Choquet-Tu stated, "It's not something where someone can easily take this idea and train a similar sophisticated attack model."

OpenAI's development of GPT-Red reflects a fundamental shift in AI safety testing as large language models become more capable and widely deployed. Traditionally, red-teaming—the practice of finding security vulnerabilities before release—has relied on human testers, but the growing complexity of LLMs and their integration with external systems like web browsers, code editors, and other AI agents has made comprehensive human-only testing increasingly difficult. The self-play loop approach OpenAI employed, in which GPT-Red iteratively attacks other models while they defend themselves, creates a training ground where both attacker and defender capabilities co-evolve. This mirrors competitive dynamics in cybersecurity, where defenders must anticipate and prepare for attacks that haven't yet been invented.

The "Fake Chain of Thought" attack exemplifies why automation may be necessary: it exploits a mechanism (chain-of-thought reasoning) that is internal to how modern LLMs solve problems, and discovering such attacks requires understanding the model's own logic. When OpenAI replicated a 2025 experiment in which human red teamers had found vulnerabilities in GPT-5, GPT-Red outperformed the human testers, suggesting that automated searching of the attack space can be more thorough in specific domains. However, the body makes clear that GPT-Red is not intended to replace human expertise—human testers can still find attacks GPT-Red misses, and vice versa, so the most effective approach combines both.

No comments yet. Be the first to share your thoughts!