Anthropic CEO: open-weight models risky, but not calling for ban

In a blog post, Anthropic CEO Dario Amodei articulated his position on open-weight models—publicly released AI systems that any user can download and run—stating clearly that Anthropic has never called for a ban on them. He acknowledges that models without dangerous capabilities are a public good. However, he maintains that capable open-weight models carry significant risks that distinguish them from closed models.

Amodei identified two primary threat scenarios. First, authoritarian states, especially China, could develop more powerful models than the US and leverage them for military superiority or enhanced surveillance capabilities. Second, powerful open models could be misused for cyberattacks or biological attacks. He notes that AI labs have observed such misuse attempts already occurring with closed models. The critical difference, in his view, is that open-weight models lack safeguards and released code cannot be recalled, creating an asymmetry in which attackers benefit more than defenders. He emphasizes that biological threats pose a particularly acute concern: sufficiently capable open models could enable rapid weaponization of viruses, whereas building countermeasures would require years even under optimal conditions. A UK AI Security Institute study supports part of this analysis, finding that open-weight models trail frontier models by a few months on cybersecurity tasks.

To address these risks without banning open weights, Amodei is advancing four policy proposals. First, he calls for stricter US chip export controls to prevent China from training powerful models without American chips. Second, he demands action against industrial distillation—a technique he says China employs to partially circumvent chip restrictions and benefit from US AI development. Third, he advocates for mandatory safety tests applicable to all powerful models regardless of their origin or licensing model, noting that recent policy proposals show growing consensus by exempting less powerful models from startups and research institutions. Fourth, he points to his own warnings about his models' capabilities as evidence of consistency, given that such warnings previously helped prompt a US ban.

Critics offer contrasting interpretations. Some argue that Anthropic's public caution about open-weight models serves its commercial interests by discouraging cheaper competition. OpenAI has similarly been reported to continue lobbying against open weights despite signing a letter—alongside many major tech companies—opposing US restrictions on Chinese open-weight models. Skeptics of Amodei's chip export and distillation controls contend that such measures primarily accelerate China's independent chip industry development and that distillation itself is a legitimate and valuable technique. Amodei's comments arrive as US authorities reportedly weigh steps to make it harder for American companies to use Chinese open-weight models, placing his position at a critical juncture in AI governance policy.

Amodei's clarification arrives amid broader debate over how the US should regulate open-weight models. His position is nuanced: he accepts that some open models create public benefit while maintaining that capable ones pose distinct risks compared to closed alternatives. His reasoning centers on the irreversibility of released models—once distributed, they cannot be recalled—and the absence of safeguards that closed-model operators can enforce. The imbalance he emphasizes is particularly stark for biological threats, where defensive measures lag far behind offensive capabilities in timeline.

The credibility of Amodei's arguments is anchored in his track record: he has previously warned about the capabilities of his own models, warnings that contributed to a US ban, indicating consistency in his risk framing rather than opportunistic positioning. However, critics interpret his stance differently, suggesting that Anthropic's public caution about open weights may also serve its business interests by discouraging cheaper open-source competition. The letter signed by major tech companies opposing US restrictions on Chinese open-weight models, alongside reports that OpenAI continues lobbying against open weights despite signing that same letter, suggests the industry's stated positions do not always align with its lobbying efforts.

No comments yet. Be the first to share your thoughts!