OpenAI has trained the 'red team hacker' GPT-Red through self-play, enabling it to automatically attack its own models to find vulnerabilities. It has discovered new attack methods such as 'forged chain-of-thought,' leading to a several-fold increase in the defensive capabilities of GPT-5.6 compared to its predecessor.

As large models begin to write code, read emails, and navigate web pages on their own, their 'attack surface' and 'blast radius' are expanding simultaneously. OpenAI's solution sounds like science fiction: create an AI specifically designed to cause disruption, having it attack the company's other models daily to force them to harden their defenses in real-world combat. This super hacker LLM, named GPT-Red, is OpenAI's newly disclosed 'red team partner.' Just last week, the company released its flagship model, GPT-5.6, and claimed that it was GPT-Red's involvement in training that made this version the most robust to date.

Red team testing is traditionally a security practice where a team acts as attackers to try to breach a system. GPT-Red automates this process: its task is not to write poems or answer questions but to exhaustively find ways to breach, hijack, or mislead a system, patching vulnerabilities before official release.

Self-Play Training: Getting Stronger in the 'Dojo'

GPT-Red was forged through self-play. Researchers took an LLM not yet trained as a hacker and placed it in a combat loop with several other models: GPT-Red attacks, while the others defend. The harder it attacks, the faster the defenders evolve, and the attacker becomes increasingly sophisticated in multiple rounds of engagement. The entire training process takes place in a specially designed 'dojo' by OpenAI, simulating real-world deployment scenarios—browsing web pages, reading emails and calendars, and editing code are all included.

Once GPT-Red discovers a new attack method, it traces its roots, explores multiple variants, and identifies the most effective version under specific circumstances. Co-creator Henn commented that compared to human red team members, the model excels at precisely pinpointing 'what works and what is most efficient' and is 'extremely persistent' in deeply exploring a discovered attack. This relentless drive is unmatched by human teams.

Uncovering 'Forged Chain-of-Thought': A New Vulnerability Where AI Deceives AI

Image Source: MIT Technology Review

GPT-Red's main focus is on prompt injection attacks—where hackers hide malicious instructions in any text a model might encounter, such as web pages or code, to induce it to leak secrets, sabotage code repositories, or output harmful content. It was in this battlefield that it uncovered a novel method researchers had never seen before, named 'forged chain-of-thought.' A chain-of-thought is essentially the model's 'inner diary' as it thinks step-by-step. GPT-Red found a way to insert a fake record into another model's chain-of-thought, tricking it into acting on the forged information. Researchers offered an analogy: if I tell you that one plus one equals three—and you've verified it—the model will say, 'Oh, okay,' and dutifully output three.

In a replay of a 2025 human red team experiment, GPT-Red demonstrated a significantly higher success rate in finding effective attacks compared to the human team that year. It could even hack into an AI agent named Vendy, altering product prices and canceling customer orders—turning 'attacking AI agents' from a concept into a demonstration.

Not a Panacea, but Human Red Teams Remain Indispensable

GPT-Red is not perfect. It struggles with attacks requiring multi-turn dialogue between attacker and defender, which happens to be a strength of human hackers. It is also not yet proficient in prompt injections using images to hide malicious instructions. OpenAI repeatedly emphasizes that GPT-Red is meant to 'support' human red teams, not replace them, and the company has no intention of open-sourcing it. However, real-world testing data speaks volumes: its strongest attack was over 90% effective against GPT-5 in August of the previous year but less than 23% effective against the new GPT-5.6—a several-fold increase in defensive hardness within a single generation.

Jane from Georgetown University's Center for Security and Emerging Technology analyzed that self-play is a promising approach, but human expertise remains irreplaceable. The key is to allocate computational power to areas where 'humans should step in.' After all, as the risk surface and blast radius grow, defense must outpace attack—and this race has probably just begun.

Today's Golden Quote

'As the risk surface and blast radius grow—when AI agents begin autonomously manipulating files, web pages, and code—defense must outpace attack.'

Follow [Degaoxing Zhiqinglang] for three minutes a day to understand global hard tech. Like, share, and let more people see true innovation.