Gaia2 benchmark evaluates LLM agents in dynamic, asynchronous environments with action-level verification

Researchers introduced Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments where conditions evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents.

Unlike prior static or synchronous evaluations, Gaia2 pairs each scenario with a write-action verifier, enabling fine-grained, action-level evaluation and making the benchmark directly usable for reinforcement learning from verifiable rewards. The benchmark is built on a consumer environment using the open-source Agents Research Environments platform.

Evaluation of state-of-the-art models shows no single dominant performer: GPT-5 (high) reaches 42% pass@1 overall but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, and Kimi-K2 leads among open-source models with 21% pass@1. Results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the 'sim2real' gap.