Kirill Brilliantov: elaborate agent harnesses add nothing

Autonomous machine learning engineering agents have made significant progress on public leaderboards. These systems are often deployed on top of increasingly elaborate machinery, including multi-agent orchestrators and dedicated retrieval subagents, motivated by long-horizon progress stagnation and limited LLM primitives. Meanwhile, the use of more primitive but improved coding agents—where the LLM has direct access to the execution environment through read, write, and bash primitives—has received little attention.

This paper fills that gap by running large-scale systematic ablation studies. The authors find that the machinery layers become redundant in the coding agent setting: under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline. They point to the backbone as the primary driver for performance and conclude that the effort spent elaborating hand-crafted harnesses yields poor returns for current MLE benchmarks.

The stakes hinge on whether these benchmark results generalize to other tasks and settings. For teams investing in agent scaffolding, the findings suggest the returns on extra orchestration or retrieval layers may be limited when the underlying model is strong, though further validation on other benchmarks would be needed to confirm this.