Jack Clark moves self-improving AI forecast to 2027 on new benchmark

P-Zero Research built TasteVal around a specific economic logic: experiment compute is scarce in AI research, so a model that picks the right experiments and reads their results correctly needs fewer runs to reach the same score. The lab's "AI Futures Model" predicts experiment compute becomes the biggest bottleneck once coding is automated, which is why the benchmark translates research taste into a compute multiplier. The top result, Anthropic's Opus 5.5 at 2.3, came with a wide statistical range of 1.15 to 4.37, meaning at the low end the model would barely lead. A single run cost roughly one-thirtieth of what an expert attempt cost on average.

The human baseline came from 24 experts who had recently worked at OpenAI, Google DeepMind, Nvidia, and Microsoft Research, with at least two people on each of eight tasks. The authors themselves admit their results may overstate how quickly research taste is improving, because the tasks can be checked quickly and cheaply — the kind of task AI labs find easiest to optimize for — and the comparison group included no top-tier researchers.

The debate around Clark's forecast has centered on exactly this capability. AI researcher Herbie Bradley argued in response to Clark's May essay that while models could plausibly take over junior researchers' work, research taste, creativity and long-term planning were nowhere in sight. A Fudan University study found AI agents contributed up to 55 percent of methodology proposals in its own model's development, but humans still made over 85 percent of final decisions, and it remains unclear how AI can generate diverse research directions and assess their value before results are in — the decision TasteVal takes off the models' plate.