New benchmark tests LLM agents' ability to coordinate — most struggle at ~6% performance

A new research benchmark tests whether large language model agents can coordinate effectively in complex, open-ended environments. The researchers evaluated 13 modern LLMs on tasks that require agents to explore worlds together, communicate with each other, trade resources, craft tools, build structures, and fight enemies.

The results reveal a stark performance gap. Most agents achieved only ~6% normalised return—a sign that coordination at scale remains difficult for current models. However, the benchmark did uncover one standout: zero-shot Gemini 3.1 Pro (run without any special training or prompting) matched the performance of the best multi-agent reinforcement learning agent that had been trained for 1 billion environment steps when tested on the hardest difficulty setting. This suggests that at least one foundation model has learned some coordination behaviors implicitly from its training data.

Beyond individual results, the research team identified coordination as a distinct problem separate from long-horizon task competence—meaning that even agents good at multi-step reasoning sometimes fail to coordinate well. In ablation tests (where they systematically disabled different capabilities), communication had the largest measured effect on team success, indicating that how well agents exchange information and align on goals is the primary bottleneck.

The full paper, project page with a public leaderboard, code, and interactive traces of agent behavior have all been released, inviting the broader research community to benchmark their own models and work toward improving LLM coordination.

The benchmark addresses a practical gap in AI evaluation: while individual LLMs have demonstrated impressive capabilities on long-horizon reasoning tasks, far less is known about their ability to work together effectively in shared environments. By designing scenarios that require exploration, resource trading, tool crafting, and collective problem-solving, the researchers created a setting where agent ability must be paired with team coordination.

The striking performance gap—most agents at ~6% normalised return—suggests that modern LLMs out of the box lack robust multi-agent coordination strategies. The fact that zero-shot Gemini 3.1 Pro matched MARL agents trained for 1 billion environment steps on the hardest setting indicates that some foundation models have learned coordination behaviors, but the majority have not. The ablation finding that communication is the largest lever is concrete and actionable: it points toward communication protocols and information exchange as the most fruitful area for improving team performance, rather than raw individual capability.

No comments yet. Be the first to share your thoughts!