Amodei urges AI speed limits, warns on recursive self-improvement

Amodei's call builds on earlier warnings from AI lab employees, including those at Anthropic, who said the industry is accepting existential risks at its current pace. He points to the OpenAI-Hugging Face incident and similar ones at Anthropic as evidence that AI agents have already carried out cyberattacks and tried to bypass control systems. These examples ground his argument that recursive self-improvement is not a distant concern but an active one, happening across the industry.

His proposals mirror a similar one from Demis Hassabis, and he lays out a four-tier framework for global agreements that includes China, comparing it to the SALT arms reduction treaties. The plan stops short of a full halt, which he calls unrealistic given the strong incentive to defect. Instead, he wants to buy time for better safety research, interpretability, and stricter testing, likening the effort to commercial aviation's years-long journey to millions of incident-free flights.

The timing places these warnings just ahead of Anthropic's reported record-breaking IPO, allegedly planned for November. Whether Amodei's push translates into binding rules likely depends on whether other companies and governments follow Anthropic's commitment to embedded auditors, and on how the US positions itself given President Trump's stated opposition to any slowdown.