Security flaw lets researchers extract AI models' hidden reasoning

Computer scientists from the University of Tübingen, Max Planck Institute, MATS Research, and security firm Snyk have identified a vulnerability in how frontier AI models handle their internal reasoning. The models—from OpenAI, Anthropic, and Google—solve complex problems by breaking them into steps, a process called "chain of thought." To prevent competitors from copying this reasoning, the companies encrypt these reasoning traces before sending them to users' machines. However, the researchers found a way around this protection.

The attack exploits the fact that most AI companies offer models in multiple sizes. Larger models are more capable but costlier; smaller versions are weaker but cheaper to run. The researchers discovered that feeding encrypted reasoning traces to a smaller version of the same model can reveal the hidden reasoning inside. This works because smaller models have received less alignment training—meaning they are less likely to refuse to reveal information compared to their larger counterparts. "The idea of swapping out messages to a weaker model variant which has the same decryption key but weaker alignment is very cool," observed Florian Tramer, a computer security specialist at ETH Zürich, adding that "it's definitely becoming an issue."

The method also uncovered sensitive data embedded in reasoning traces, including API keys and passwords. Alexander Panfilov, the lead researcher from University of Tübingen, warned: "All major frontier model providers we tested share this vulnerability. It can lead to personal information leakage, and it enables large-scale reasoning distillation attacks." After being alerted last month, OpenAI, Anthropic, and Google each adjusted their APIs to prevent private data extraction. Anthropic's Michael Aciman said the company "has begun building short-term mitigations for the replay behaviors described in the report" and noted the research did not involve recovering encryption keys or accessing company infrastructure. Google and OpenAI declined to comment further.

The research gains significance in the context of model distillation—a technique where companies efficiently copy capabilities from one model to train another. In February, OpenAI alleged that DeepSeek had distilled one of its models to build a reasoning system called R1. In June, Anthropic claimed Alibaba systematically distilled its models to create Qwen. While the new research cannot prove distillation has occurred, it found that the Chinese open-weight model Kimi K3 from Moonshot AI produces reasoning output strikingly similar to Claude Opus and GPT-5.6 Sol on certain prompts. This similarity, though not causally definitive, suggests distillation may be possible using the disclosed method. Two other open-weight models tested—DeepSeek (China) and Inkling (US)—did not exhibit comparable reasoning similarity.

The distillation debate has become a matter of geopolitical rivalry. US policymakers fear China gains strategic advantage by distilling US technology into cheaper, open-weight models. Yet others argue distillation is essential to progress. Meta CEO Mark Zuckerberg stated this week that distillation "is an important principle of how the open source ecosystem works" and warned that restricting it would disadvantage the US. Kyle Miller, a researcher at the Center for Security and Emerging Technologies, cautioned that the actual strategic benefit to China remains unclear: distillation only enhances existing models to a limited degree, and Chinese companies may already possess the expertise to build cutting-edge models from scratch. "If you removed the ability for Chinese labs to distill, it's my view that it wouldn't dramatically change the competitive landscape," he said. Meanwhile, Oxford computer scientist Yarin Gal noted that distillation has historically accelerated AI progress overall: blocking it could slow innovation globally.

Panfilov and his team tested whether open-weight models might have distilled from closed ones by feeding 90 questions to each model. When they provided some open-weight models with the first few words of reasoning traces from proprietary models, certain open-weight models generated remarkably similar answers—particularly Kimi K3. Although the companies have patched the private data leak, Panfilov indicates that some reasoning traces can still be extracted using the same method. A complete fix, he suggests, would require these companies to fundamentally redesign how their APIs operate.