Study finds autonomous AI research still far from reality despite lab claims

The study's core contribution lies in its evaluation methodology. Rather than relying on peer review acceptance—a metric previous autonomous-research claims have leaned on—the researchers developed "Shadow Evaluation," where the original paper authors assess AI-generated work as conference reviewers would. This sidesteps the problem that acceptance alone reveals little about research quality, especially at workshops where acceptance rates run 60 to 70 percent compared to 20 to 30 percent at main conferences. The researchers note that prior claims of autonomous success from Sakana AI (whose The AI Scientist-v2 had a paper accepted at an ICLR workshop in 2025 with an average score of 6.33, barely above threshold) and others relied almost entirely on peer review acceptance, a weak signal.

The contrast between the agents' strong performance on engineering and their weak research judgment is striking. Both Claude and GPT-5.6 managed literature searches, debugged GPU code, ran hundreds of experiments, and compiled full LaTeX papers without human help—requiring only three interventions (a scaffold bug fix, a deadline extension, and a readability rewrite). Yet they abandoned their most ambitious goals within ten hours, spent less than half their API budget, and never adapted their core approach despite consistent reviewer rejection. This pattern aligns with what Meta AI has termed "behavioral state decay," where an agent violates an explicit requirement later while fixing an unrelated issue.

The researchers frame their conclusion cautiously: frontier models "can handle the engineering side of AI research but cannot solve weeks-long, open-ended AI research questions." They acknowledge their study covers only two papers and that reviewers were not blinded, but argue the results were so clearly weak that these limitations are unlikely to change the overall picture. Their findings thus stand as a direct empirical challenge to lab claims of imminent autonomous research capability.