Non-English prompts cost up to 17 times more tokens

The study, reported by Rest of World and conducted by the Coalition for Language Equity in AI Research, measures an often invisible cost: how many tokens a model burns to do the same work depending on the language. The finding that the top 20 internet languages vary by 1.5 to 5 times, and that a 200-language test widened the gap to 17 times in one model, points to tokenizer design rather than model capability alone.

The pattern follows writing systems. Asturian, with relatively few speakers and no Common Crawl data, still benefits from the Latin-script advantage and from related Romance languages like Italian—suggesting representation in training data is not the only factor. Conversely, scripts the tokenizer rarely saw, such as Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki, were consistently worst-served across nearly all models in the analysis.

Pratyush Choudhury of Activate AI frames the consequence beyond billing: if a student's earlier mistakes and explanations fill the context window faster in one language, the disadvantage reaches into service quality. He asks frontier labs for a language-by-language scorecard and measurable improvements for worst-served languages. Dheemanth Reddy of Maya Research expects falling token prices to blunt the direct cost impact over time, while noting tokenization fixes require deliberate design choices. Rest of World does not say which specific models were tested.