Vals AI: agent teams cost 5.1x more, barely beat solo agents

The finding lands against a backdrop of rising enthusiasm for multi-agent architectures, where teams of AI models are pitched as a way to tackle harder problems. Vals AI's benchmark, run on its Vibe Code Bench with GPT-6 Sol and Claude Opus 5.5, tested that premise directly by comparing each model solo and in teams at medium and maximum reasoning effort. The cost gap was wide — 1.8x to 5.1x — and the quality gap was mostly invisible, with only GPT-6 Sol at medium reasoning showing a statistically significant improvement of 7.3 points.

Anthropic's own data with Opus 5.5 points in the same direction. In its tests, larger teams reached a given performance level faster, but going from ten to 100 agents only nudged scores up slightly after 24 hours. On the knowledge base task, the score moved from 0.53 with one agent to 0.74 with 100 agents — a gain that came with far more tokens. Fable 5.1 showed stronger quality gains on the Lean theorem proving task above ten agents, yet still scored below Opus 5.5 across all tests, and its knowledge base score dipped slightly from 30 to 100 agents.

The practical caveat comes from OpenAI researcher Noam Brown, who told the Dwarkesh Podcast that the benefit depends heavily on the task: web research and math parallelize well, but writing a novel does not, and throwing 10,000 agents at a novel would be as pointless as throwing 10,000 people at it. OpenAI developer Eric Provencher has warned that agent swarms are most likely wasted money because coordination between agents breaks down — a cost he called the coordination tax.