AI evaluation costs have become a major bottleneck as benchmarks move from static tests to agent-based evaluations, with single runs now costing thousands of dollars and compression techniques losing effectiveness.

The Holistic Agent Leaderboard spent about $40,000 to run 21,730 agent rollouts across 9 models and 9 benchmarks. A single GAIA run on a frontier model can cost $2,829 before caching. Exgentic found a 33× cost spread on identical tasks depending on scaffold choice (the specific instructions and structure given to an AI agent).

Static benchmarks like HELM were compressible by 100× to 200× through techniques like Flash-HELM and tinyBenchmarks, which found that model rankings could be preserved using far fewer test items. Agent benchmarks are messier: cost varies by four orders of magnitude across different tasks, and compression yields only 2× to 3.5× reductions while preserving rank fidelity.

In scientific machine learning, evaluation costs now exceed training costs by roughly two orders of magnitude. The Well requires 3,840 H100-hours for a full four-baseline sweep and 960 H100-hours to evaluate a single new architecture, reversing the traditional deep-learning model where training dominated compute budgets.

Pricing variations between models create cost spreads independent of accuracy gains. Claude Opus 4.1 charges $15 per million input tokens and $75 per million output, while Gemini 2.0 Flash charges $0.10 and $0.40—a two-order-of-magnitude spread on input alone.