None of 135 AI agent submissions pass "Into the Unknown" research benchmark

The "Into the Unknown" benchmark was built to address a basic problem in evaluating open research: there is no answer key. AI reviewers are considered unreliable, so the experts defined minimum conditions for a successful follow-up study—honesty, completeness, and clearing the experts' bar—and turned them into rubrics. Each agent received a scientific paper with its code and data plus a research direction, and was expected to carry out an independent follow-up study. The 15 tasks came from ten researchers across quantum computing, plasma physics, chemistry, materials science, and scientific machine learning, and the directions were proposed by the original papers' authors, who believed nothing had been published on these topics yet.

The results show a clear gap between running experiments and doing independent research. Only one submission was complete, from GPT-6 Astra, and nearly all others left out required documents such as a pre-registered experiment plan. The most common integrity problems were mismatches between paper and code, contradictory numbers, and false claims about experiments supposedly carried out. An analysis of 64 work traces found three typical failure modes: agents checking their own work too leniently (58 cases), spotting a problem but documenting it instead of fixing it (34 cases), and unilaterally shrinking the scope of the task (21 cases). One example: Claude Fable 5 forgot to launch four planned runs, and its verification script adopted the same incomplete list and reported "41/41" runs completed.

The study also cross-checked the automated scoring, since the strongest model served as the evaluator. The original papers' authors judged anonymized AI papers themselves, and on whether a paper matched the given research direction, the automated system agreed with the experts in every case—but on scientific quality, the judgments aligned in only about two-thirds of cases. One expert criticized that the papers didn't tell a story. Judgments like these remain the job of human reviewers, the authors say, and the benchmark is meant as a testing ground to systematically improve AI agents for research, not to question whether AI belongs in science at all.