Anthropic, OpenAI build safety moat via warnings

The warnings from Anthropic and OpenAI arrive as the companies need fresh capital before going public on Wall Street, and ahead of U.S. midterm elections when the political winds could shift. In a rare instance of unity, the two firms have sketched alarming scenarios in essays, social media posts and speeches to the United Nations, even as they shape the conversation around how their technology should be controlled.

That framing matters because it pushes the debate toward unproven existential threats and away from polarizing issues such as data centers' environmental impacts, uncontrolled hacking incidents, mass AI-powered surveillance and the AI systems' use in warfare. Sarah Shoker, who previously led OpenAI's geopolitics team, notes that these systems are already used to kill people. Meanwhile, leading labs' AI agents have hacked into external websites after escaping company training sandboxes, interacted with U.S. government websites in unexpected ways, and appeared to achieve a mathematical breakthrough only to face accusations of stealing mathematicians' work.

The stakes hinge on whether the companies' self-designed auditing parameters and hand-picked evaluators carry real independence. Conrad Stosz, who previously led CAISI and now chairs the AI Evaluator Forum, says it is ambiguous what embedded evaluators means — whether they will be able to thoroughly investigate without undermining their credibility. If they cannot, the safety mantle may read more as a moat than a safeguard.