AI agents fail at creative research, raising doubts on self-improvement timeline

The promise of recursive self-improvement—where AI systems accelerate their own development—has become the AI industry's boldest near-term milestone. Both OpenAI and Anthropic have publicly charted progress toward this goal. In July, OpenAI highlighted that GPT-5.6 Sol helped post-train a smaller model, saving weeks of work. Anthropic published a blog post in June titled "When AI Builds Itself," signaling confidence in the trajectory. However, the Princeton study exposes a critical gap between what AI agents can do narrowly and what they can do creatively.

The researchers designed "shadow evaluation," a new testing method, to measure open-ended research capabilities—the kind of work that cannot be reduced to yes-or-no answers. Their finding that Claude Opus 4.8 could manage all the engineering legwork (literature review, experiment running, result compilation) but produced papers rejected for lack of originality points to a deeper limitation. Sayash Kapoor, one of the study's leads, attributed this to how AI models are trained: they excel at tasks that can be checked automatically through reinforcement learning, but open-ended work requires training environments that do not yet exist. Anthropic cofounder Jack Clark later echoed this observation, noting the company found similar constraints when attempting to automate AI safety research—a domain also demanding intuitive judgment over rote execution.

The study has limits. It evaluated only two papers, and knowing their graders were human researchers evaluating AI work may have influenced their judgments. The researchers themselves had discretion in study design, introducing potential bias. Yet the findings align with what AI companies report internally and raise a structural question about the path to self-improvement: can AI progress through grinding improvements on narrow, measurable tasks alone, or does recursive self-improvement require the creative leaps—like the invention of transformers—that Kapoor argues have driven the field's biggest breakthroughs? That, Kapoor concludes, is "frankly the trillion-dollar question right now."