AI refusal: a safety wall nobody fully understands

The idea of AI refusal traces back to a 2021 paper by a team at Anthropic, which argued large language models should be helpful, honest and above all harmless — meaning an AI asked to aid in a dangerous act should politely refuse. That principle is now standard. But the piece describes how the earliest models did not refuse at all: former OpenAI safety researcher Steven Adler said the company's early models would "blab on about anything," and Harvard's Ryan McBain recalls early chatbots readily generating responses to questions about suicide.

The mechanics of teaching refusal are stranger than they look. When OpenAI prepared to release ChatGPT in 2022, it enlisted dozens of red-teamers, including Paul Röttger, to probe the model; Röttger found that when he asked it to write a recruitment post for Al Qaeda, it complied, and a few months later it said no. Refusal is a set of activations that light up among a model's billions of parameters, and a recent Google-funded study suggests refusal behavior appears in activation space as "high-dimensional polyhedral cones."

Because refusal is unreliable, companies stack smaller classifier models around their systems in what the industry calls the Swiss cheese model — slices riddled with holes, on the theory that enough layers form an impenetrable rampart. The cost of this: Anthropic said one classifier added 24% to its chatbots' compute costs. It also describes the trade-off at the core of the problem, quoting current OpenAI employee Dillon Bowen, speaking in a personal capacity, on the industry "trying to do two things at once": democratizing AI's benefits while stopping malicious actors from using those same capabilities.