Anthropic researcher Jacob Coxon resigns over AI 'gambling with our lives'

Coxon's resignation highlights a growing divide within the AI community over the safety of rapid advancement. His specific concern, recursive self-improvement (RSI), points to a future where AI could improve itself without human intervention, a scenario that some researchers believe could lead to catastrophic outcomes. This is not an isolated worry; the article notes that Anthropic and OpenAI are already using AI to automate model development, and recent advances include OpenAI solving a major math problem and Anthropic using Claude to complete a proof in 11 days.

The reaction from within Anthropic is notable: Evan Hubinger, the alignment science lead, publicly affirmed Coxon's concerns, stating he personally thinks there is a >10% chance AI kills all humans within the next decade. This internal validation amplifies the seriousness of the issue, moving it from fringe speculation to a topic of earnest discussion among those building the technology.

The stakes are high for Anthropic as it approaches an IPO. Investor Azeem Azhar's tweet suggests that if Anthropic is intellectually honest, the risk of AI should be disclosed in its S-1 filing, which could become a litmus test for the company's credibility. The political response, including proposed legislation from Bernie Sanders and Lori Trahan, indicates that these concerns are beginning to translate into regulatory action, which could have significant implications for the entire AI industry.