METR introduces 'expenditure horizon' metric for comparing AI and human work costs

METR's new 'expenditure horizon' metric addresses a fundamental question in AI research: at what point does deploying an AI agent become more cost-effective than having a human tackle the same problem? The metric emerged from an observation METR has made in prior testing: AI agents often excel at simple, low-cost tasks but fall behind as budgets grow and tasks become harder. By converting all costs—including compute, experimental overhead, and human labor—into a single currency, the metric avoids the misleading simplicity of binary pass-or-fail benchmarks and instead reveals how much improvement each dollar buys.

The NanoGPT speedrun proved an ideal testing ground because it isolates a single, well-defined optimization problem with a measurable outcome: reducing training time on standardized hardware. Since May 2024, community contributors have achieved a cumulative 33× speedup across 82 documented steps, with training time dropping from about 45 minutes to under two minutes. METR's cost analysis—based on interviews and AI-generated effort estimates—found humans invest roughly $2,500 per one-percent improvement, totaling about $250,000 of combined human effort. By contrast, the six AI models METR tested (GPT-5, GPT-5.5, Opus-4.1, and Opus-4.8) generated expenditure horizons in the low four figures at most, with only the two most capable models delivering measurable progress. This suggests that current AI agents, even when given substantial compute budgets, remain far from matching human researcher productivity on open-ended optimization tasks.

However, the study carries important caveats. METR tested only older models released before Anthropic's Opus 5 and other newer systems, which claim significant efficiency gains (Opus 5 uses 26 percent fewer compute steps than its predecessor while achieving higher performance). More fundamentally, the metric measures AI operating in isolation—a scenario that rarely occurs in practice. Real researchers use AI as a collaborative tool, and METR sketched a hypothetical 'hybrid curve' suggesting that humans deploying AI wisely should outperform both pure human and pure AI approaches. Yet METR's own prior work has shown human-plus-AI setups sometimes underperform humans alone, suggesting the added value depends heavily on how the AI is deployed. A controlled experiment—comparing the same researchers with and without AI support—could resolve this question but would be logistically challenging to organize.