Anthropic's Evan Hubinger puts 10-year AI extinction risk above 10%

The posts follow Jacob Coxon's departure from Anthropic, whose claims Hubinger said were right and whose thread Marks called worth reading. Hubinger drew a line between current models, which he said the company's August risk report already described as low risk, and superintelligence arising from recursive self-improvement, which he said is moving faster than expected — pointing to a June blog post on that topic.

Marks framed the situation in five points: developers believe their own technology could cause human extinction or something comparably bad within a few years, and the more senior the employee the stronger the concern tends to be; commercial motives plus competition with less cautious firms keep development going; AI cannot be programmed for desired behavior the way conventional software can, and serious misbehavior happens often; there is no method for robust alignment, only for steering toward better behavior; and many employees want to slow down.

What the outcome hinges on, in this reading, is whether the safety staff pressing for slower work can match their concern with a concrete alignment plan. Hubinger's own post suggests that plan does not yet exist at Anthropic and that the company is not clearly on track to it, while Marks signals he stays because he expects safety research can lower the odds of the worst outcomes.