Alibaba's CommerceAgentBench: AI agents complete 61.7% of real tasks

Alibaba.com's Accio team built CommerceAgentBench from its own operational data—10 million active small-business users, 1.6 million real conversations, and 200,000 execution traces—to grade AI agents on whether work gets done correctly in the system where it lives. This approach reflects a view that commerce requires finished outcomes, not articulate responses, since a wrong product or a missed fraud signal matters more than how polished the agent sounded.

The test's results show real progress and real limits. The strongest model completed 61.7% of tasks, enough to automate some routines but not enough to remove human oversight. Failures concentrated in areas like fraud detection, landed-cost calculations, and multi-leg shipping routes—tasks where errors can ripple through supply chains at scale. Notably, no single model dominated; leadership rotated by task category, suggesting that model choice should follow the specific job rather than general benchmarks.

The stakes for businesses hinge on adopting 'precision delegation'—using pass rates to decide which workflows to hand over and which to keep human-checked. As more businesses run similar agents, individual mistakes could become correlated across thousands of operations, multiplying errors in listings, missed fraud, or routing failures. Open benchmarks like this one let buyers verify vendor claims, but each industry will need its own tests built by people who understand what a bad outcome costs there.