AI agents fail open-ended research despite engineering prowess

The research directly challenges a core assumption in forecasts of rapid AI progress: that frontier AI agents can automate AI research. While Anthropic and OpenAI have made public claims about AI systems contributing to research—including OpenAI's statement that GPT-5.6 Sol helped with post-training and saved researchers multiple weeks—the evidence base for agent-driven research capability has been thin. Most prior evaluations focus either on narrow, verifiable tasks with fixed metrics (like optimizing a known benchmark) or on submitting AI-generated papers to peer review, which the authors argue is stochastic and reveals little about how many attempts failed before acceptance.

The shadow evaluation method introduces a third approach: pairing agents with unpublished papers, asking them to answer the same central research question the authors did, and having those authors grade the result. This design avoids contamination (the agent cannot look up published findings) and ensures expert grading on questions they have spent months thinking through. The result is unambiguous: agents can run experiments, write code, and execute a research workflow, but they cannot judge when to stop pursuing a line of inquiry, creatively respond to evidence their approach is failing, or allocate their resources toward the most promising directions. Both agents finished with hours of wall-clock time and less than 50% of their budget remaining, despite papers that did not meet their own success criteria.