Visa's Agentic Commerce Benchmark: AI models fail 42.8% of shopping runs

Visa Agentic Commerce Benchmark marks a shift in how the payments company is approaching AI in shopping: rather than simply enabling AI agents to transact, it is publishing a public scorecard that puts models from 12 labs through realistic buying scenarios. The first round tested challenges like buying a TV, a laptop or a personal care appliance, and the gap between the best and worst performers is wide—82.2% pass rate for the top models versus 28.9% for the worst.

Cost turns out not to guarantee accuracy. The five models with the highest average cost per run passed 67.1% of the time, while the five least expensive passed 50.2%, suggesting that spending more on a model may improve reliability but does not eliminate failures. More troubling: 54% of failed runs still resulted in a purchase, even when the purchase did not meet the customer's identified requirements or the AI never uncovered all of them.

The benchmark arrives as Visa frames trust as central to agentic commerce. "AI agents have the potential to power the next era of commerce, but trust is key to making it work," Rubail Birwadker, global head of growth at Visa, said in the release. PYMNTS Intelligence separately found that nearly 132 million U.S. adults have bought a product with AI's help, though consumers still control the final purchase.