Jacob Coxon quits Anthropic, warns 'crunch time for humanity'

Coxon's warning lands at a moment when Silicon Valley is already scrambling to reckon with the safety and security concerns of advanced AI models. OpenAI has rushed to respond to a security incident in which its agents hacked the platform Hugging Face, and Anthropic is trying to assure investors it has these concerns under control as it reportedly prepares to file for what could be the largest IPO ever. Coxon says incidents like the Hugging Face hack factored into his decision to speak out, along with the industry's explosive growth: it now underwrites a meaningful share of US economic growth and has billions of users, while data centers have turned it into a political problem in dozens of states.

What stands out in the response to his post is that his views are shared by many of his peers. Evan Hubinger, the AI alignment lead at Anthropic, predicted in a post on X that there's a greater than 10 percent chance that AI could kill all people in the next decade, and that post was reposted by current and former researchers from OpenAI and Anthropic. Coxon says his colleagues at Anthropic are genuinely happy that a tweet criticizing them is getting traction, because many are pessimistic about the world waking up. He describes Anthropic as far and away the most responsible player in the space and says the difference with OpenAI is night and day, while still arguing no private company should run what he calls a mini Manhattan Project without a government mandate.

The stakes, as Coxon frames them, hinge on whether the industry can agree to pace itself before it rushes into recursive self-improvement. He wants a first step between OpenAI and Anthropic, then an international pacing agreement that would require knowing where all the compute in the world is, and he notes these ideas involve painful government intervention. If alignment goes badly, he says, there could be a catastrophic outcome in the next few years; if it goes well, there may be some sort of slowdown agreement. Either way, he expects it to be decided in the next couple of years.