Ai2: small prompt tweaks swing GPT-5.4 novelty calls 93.5%→40.9%

Ai2's study sits inside a broader push to automate parts of scientific discovery. The lab already runs DataVoyager for structured-data exploration, AutoDiscovery for generating and testing hypotheses, Theorizer for synthesizing evidence into candidate theories, and Asta alongside them. Those systems can propose ideas, but judging whether an idea is genuinely new is typically handed to another LLM — a novelty judge whose model, prompt, evaluation format, reasoning budget and retrieval setup are chosen ad hoc by each team.

The benchmark's construction reflects that concern about validation. Instead of treating paper acceptance as a proxy for novelty, Ai2 mined ICLR 2026 reviews for passages where reviewers explicitly praised or disputed originality. Accepted papers in the top 10% of their area whose reviewers positively identified novelty, with no dispute, formed a 154-idea high-novelty pool; rejected papers in the bottom 10% with originality concerns and no positive novelty signal formed a 145-idea low-novelty pool. An annotator agreed with the extracted signal polarity in 47 of 50 cases.

A second setting paired the 154 human ideas with 154 produced by a simple LLM generator, with the caveat that the generated ideas are assumed less novel — weak labels, not verified judgments. An expert who reviewed 30 such pairs, weighted toward cases the strongest judge got wrong, preferred the human idea in all 30 and judged none of the generated ideas novel. Ai2 notes that many core findings do not depend on the labels at all: if two reasonable evaluation setups give opposite answers for the same pair of ideas, that disagreement itself reveals instability.