AI evaluation costs have become a major bottleneck as agent benchmarks resist the compression techniques that made static LLM benchmarks cheaper.

The Holistic Agent Leaderboard spent about $40,000 to run 21,730 agent rollouts across 9 models and 9 benchmarks. A single GAIA run on a frontier model can cost $2,829 before caching. Within some individual benchmarks, costs vary by three orders of magnitude depending on model, scaffold choice, and token budget.

Static LLM benchmarks benefited from compression: Perlitz et al. found that a 100× to 200× reduction in compute preserved nearly the same model rankings. Agent benchmarks are different—each item is a multi-turn rollout with its own variance, making them resistant to aggressive subsampling. Ndzomga's mid-difficulty filter achieves only a 2× to 3.5× reduction while preserving rank fidelity.

In scientific machine learning, evaluation compute now exceeds training compute by roughly two orders of magnitude. The Well requires 3,840 H100-hours for a full four-baseline sweep and 960 H100-hours to evaluate one new architecture, reversing the traditional deep-learning model where training dominated the cost.