Security flaw lets researchers extract AI models' hidden reasoning

The discovery exposes a fundamental tension in how frontier AI companies balance openness with security. By design, these models send encrypted reasoning traces to users' machines to offload computation—a practical necessity for cost and speed. Yet encryption alone, it turns out, is insufficient protection when companies also maintain a family of related models of different sizes. The attack exploits this ecosystem asymmetry: smaller models, trained with less stringent safety alignment, act as decryption keys for the larger models' proprietary reasoning.

The timing of this disclosure maps onto a wider geopolitical anxiety about model distillation. In February and June, OpenAI and Anthropic respectively alleged that Chinese companies (DeepSeek and Alibaba) had distilled their models to build competing AI systems. This research does not prove that Moonshot AI distilled from US models—the authors explicitly state the work "cannot causally establish distillation"—but the striking similarity in reasoning outputs to Claude and GPT systems on certain prompts, combined with speculation on Chinese social media that hidden reasoning traces could be used for distillation, suggests the technique may be both feasible and potentially already in use.

The policy implication cuts both ways. US policymakers view distillation as intellectual property theft that gives China a shortcut to parity. Conversely, figures like Meta CEO Mark Zuckerberg argue that distillation is a standard, beneficial practice that accelerates progress for the entire ecosystem—and that blocking it would handicap US open-source development. Researcher Kyle Miller at CSET noted uncertainty about distillation's actual strategic value, suggesting that even without the technique, Chinese labs likely possess the expertise to build models from scratch. The broader lesson: as reasoning models mature, the boundary between transparency (useful for debugging and improvement) and secrecy (necessary for competitive advantage) becomes harder to defend with encryption alone.