Cheap AI models cost more in 32％ of tasks, study finds

The study, run by researchers from Stanford, Carnegie Mellon, UC Berkeley, and Microsoft Research, tested 8 cutting-edge AI models on over 6,800 tasks spanning math, programming, and science. Its central finding is counterintuitive: a lower-priced model does not automatically translate to a lower bill. In one example, the pricier Gemini 3.1 Pro completed a YouTube frame-by-frame analysis task in 85 steps for 1 dollar, while the cheaper Gemini 3 Flash ran over 1,000 steps, failed to finish, and cost 14 dollars.

The researchers attribute this to AI 'overthinking' and 'overacting.' On complex tasks, weaker models tend to burn more reasoning steps and actions, so a stronger model can finish faster and cheaper. A separate example saw Gemini 3 Flash consume over 60,000 thinking tokens while the more capable GPT-5.4 solved the same problem in 25 tokens.

The findings also show that cost is not even stable within a single model: running the same instruction repeatedly produced a most-expensive run 9.7 times the cheapest. Lead author Lingzhao Chen's point that price alone should not decide which model is cheap suggests the real stakes fall on teams comparing AI costs, whose budgeting may hinge on whether they evaluate per-task outcomes rather than sticker rates.