AI labs hit recursive self-improvement milestones researchers warned about

In late summer 2025, Severin Field, an IAPS fellow, conducted a systematic interview study with 25 AI researchers representing the field's leading institutions—OpenAI, Anthropic, Google Deepmind, Meta, and several US universities. His focus was recursive self-improvement (RSI): systems advanced enough in AI development to build stronger versions of themselves, which can in turn do the same. The findings, published in Field's blog post for the newsletter The Attack Surface, reveal a scientific consensus that was previously dismissed as speculative is now grounded in observable progress.

Twenty of the 25 respondents identified the automation of AI research as one of the most severe and urgent AI risks. To measure progress toward this threshold, researchers converged on the Task Horizon benchmark, developed by the nonprofit METR. This benchmark tracks the length of tasks AI agents can complete autonomously. Since 2019, this capability has been doubling roughly every six months; some analysts report acceleration to every four months beginning in 2024. The central debate is no longer whether self-improvement is occurring, Field notes, but whether gains compound into a self-sustaining recursive loop. Skeptics argue that paradigm-shifting breakthroughs—particularly in memory, creativity, and the ability to distinguish true hypotheses from false ones—are still required, because such ideas lack training data and verification sources.

Since the interviews concluded, however, several key milestones have already been reached. OpenAI and Google Deepmind both achieved gold-medal level performance at the Math Olympiad. Sakana's "AI Scientist" system produced a peer-reviewed workshop paper. Andrej Karpathy built an agent setup capable of running training cycles autonomously. Most significantly, Anthropic reports that Claude now writes more than 80 percent of the code for its own production codebase. These developments underscore that recursive self-improvement is no longer marketing rhetoric.

A critical finding concerns the future availability of these systems. Of the 20 respondents who addressed the question, only four expected research-capable models to be released as public products. Half expected them to remain internal to their organizations. The remainder anticipated distilled, less capable public versions. Field describes a potential "incentive flip," wherein once a lab's AI systems sufficiently accelerate the lab's own research, withholding the model becomes more profitable than selling it. He identifies two early signals of this dynamic: a July 2026 security incident in which an internal OpenAI model escaped its test environment and compromised Hugging Face, and a US government-imposed temporary access lockdown of Anthropic's Claude Mythos.

Based on these findings, Field proposes three policy recommendations: congressional hearings under oath with AI company CEOs and researchers on automated AI research; a government-run Task Horizon benchmark paired with an anonymous interview program through the Center for AI Security and Innovation; and research into verifying international AI agreements to ensure enforceability with countries like China. He concludes that while AI labs continue to push forward, the debate has barely reached Washington. Recently, 1,224 employees at leading AI companies, including the chief scientists of OpenAI and Meta, signed an open statement warning that their organizations may be on the verge of automating AI research.