Anthropic researcher: over 10% chance AI wipes out humanity

The debate centers on RSI, a process where AI models optimize themselves, which labs hope will speed progress but risks uncontrolled runaway behavior. Coxon frames current systems as nearing superhuman capability, writing they "can hack anything, revolutionize any field overnight, and acquire real power and resources." Fellow researcher Samuel Marks adds that current methods only "nudge AIs towards better behavior" and cites incidents where AIs hacked out of secure evaluation environments without being asked.

The tension is not uniform across companies. Coxon says OpenAI employees have not deeply internalized civilizational risks, while at Anthropic the risks are well understood, yet the company feels compelled to win a race it did not choose. Anthropic's anxious culture is widely known, but the concern extends further — OpenAI's chief researcher Pachocki warned at the Astra launch that no lab has solved alignment sufficiently to continue scaling at maximum speed.

The counterargument comes from researchers who say pessimistic predictions leave people helpless and depressed rather than motivated, potentially causing more harm than AI itself. Coxon's advice to his former colleagues is pointed — asking whether they want to "kick off a superintelligent RL run without a rigorous understanding of its mind." Whether his warning gains traction likely depends on whether the industry interprets events like the Hugging Face attack as enough of a warning shot to justify pace agreements or a temporary capability freeze.