Chinese AI models excel at coding tasks, but pricey flagship planning beats cheap claims

Moonshot's Kimi K3 landed this week with a million-token context window and the familiar refrain: Chinese models are now the cheapest way to run AI. To test that claim, one developer ran the same real-world job—building a small leads CRM database on Supabase with three tables, security policies, triggers, and seed data—through eight frontier models using identical specifications. The only criteria: passing tests (supabase db reset && supabase test db returns green) or it didn't count.

The key insight was recognizing that an AI coding loop has two distinct roles: a planner reads the spec once and breaks it into tasks, while a doer types through every task and writes actual code. By swapping planners while keeping the same "doer" model constant, the benchmark isolated planning cost separately from execution cost. Kimi K3 emerged as the expensive planning option: at $3 input / $15 output per million tokens, its single planning call cost 46 cents—thirteen times what a cheaper Western alternative (luna) cost—because "thinking" tokens bill at the full $15 output rate. Even smoke-test calls asking for single-word replies ran 51 cents total. Despite writing the most detailed task notes of the entire field, Kimi K3 landed mid-table on cost per passing test.

DeepSeek V4-Pro took the opposite trap: it offered the cheapest plan among reasoning-tier models but produced the thinnest output, generating 43 tests where rivals produced 101. The honest Chinese value pick for planning was DeepSeek V4-Pro at $0.65 with 83 passing tests, yet even that was beaten on value by Grok-4.5, an American model. Claude Opus showed a third trap: lowest bill overall, decent cost per test, but only a third of the coverage—its plan asked for less.

The story flipped entirely on the execution side. When the identical plan was routed through different "doer" models, DeepSeek's flash model passed every gate at prices that round to nothing—nineteen times cheaper than comparable Western alternatives for identical green tests. For the code-writing, test-grinding phase of AI coding, Chinese models delivered genuine cost leadership that was not close.

Three invisible costs explain why price lists and leaderboards tell different stories. First, reasoning models bill for hidden "thinking" tokens at full output rate, creating a gap between advertised and real cost; Kimi K3's smoke tests proved this painfully. Second, several benchmark stars—Kimi K3, GLM-5.2, DeepSeek V4-Pro, and the GPT-5.6 family—top coding leaderboards but return empty results when run through developer tools like the Claude Code CLI, because their answers arrive wrapped in reasoning blocks the tooling discards. A model that returns nothing in your harness costs everything. Third, on cheap aggregator routes, the provider behind a model ID can change mid-day; cheap requires re-verification per run, not per month.

The winning formula from eight rounds of testing: pair a frontier reasoning model for planning (Grok-4.5 or GPT-5.6 luna, both under a dollar all-in) with DeepSeek's flash model for typing everything, using the test gate to keep it honest. That combination produced roughly 100 passing tests for under a pound. The all-premium version of the same run cost $7.41 and produced fewer tests. So Chinese AI is the cheapest for execution—emphatically, 19-to-1 yes—but for planning, the newest Chinese flagships price like Western ones and lose on value. The lesson: buy the roles separately, measure cost per verified output instead of cost per token, and smoke-test through your own tooling before trusting anyone's leaderboard.

The benchmark test isolates a critical flaw in how AI pricing and performance are typically compared. Leaderboard rankings and per-token pricing lists obscure the real cost structure of AI coding: a planning phase that reads a spec and breaks it into tasks, followed by execution phases that grind through actual code-writing. By measuring these jobs separately across eight frontier models, the benchmark reveals that Chinese competitive advantage is narrowly situated in execution, not planning. Kimi K3's expensive "thinking" tokens and thin reasoning output from GLM-5.2 undermine the claim that Chinese models are universally cheaper. The true cost winner pairs a Western planning model (Grok-4.5 or GPT-5.6 luna, under a dollar all-in) with DeepSeek's flash model for typing, yielding roughly 100 passing tests for under a pound—outperforming the all-premium version at $7.41 with fewer results.

The hidden costs amplify this disparity. Reasoning models levy full output rates on invisible "thinking" tokens, creating a gap between advertised and actual pricing that leaderboard users never see. More damaging, several benchmark stars fail to work through standard developer tools—the Claude Code CLI filters out reasoning blocks, leaving users with empty returns despite paying for full computation. On cheap aggregator routes, provider identity can shift mid-day under the same model ID, forcing continuous re-verification rather than monthly trust. Together, these traps mean that list-price comparisons and leaderboard rankings mislead: the honest comparison requires smoke-testing each model through your own tooling, measuring cost per verified output (tests passed, not tokens consumed), and separating reasoning from execution before deciding whether a model is truly cheap.

No comments yet. Be the first to share your thoughts!