GPT-5.6-terra uses 48.5% more tokens than Mimo-2.5-pro in agent tasks

The developer began testing GPT-5.6-terra after OpenAI provided expiring free credits, using it as their primary model for several days before deciding to quantify its efficiency relative to xiaomi/mimo-v2.5-pro, which they had used previously. To ensure a fair comparison, they extracted transcript data from 95 Terra tasks and 81 Mimo tasks drawn from the same repository with similar task distributions, using Dirac as the harness and measuring only final transcript size to eliminate caching effects (Mimo cached significantly better than Terra, according to the developer's note).

The headline result: GPT-5.6-terra's average transcript of 91,794 tokens was 48.5% larger than Mimo's 61,831 tokens. Drilling into tool usage revealed why. On read_file calls, Mimo made 12.68 per task but read disciplined line ranges; Terra made only 6.99 calls per task but read at least a few hundred lines each time, resulting in more total tokens consumed. On execute_command, the difference was even starker anecdotally—Terra executed commands like npm run test:unit 2>&1 (full output), while Mimo typically filtered with npm run test:unit 2>&1 | tail -20. Terra also showed a pattern of edit-and-check loops for trivial changes, making 3.74 edit_file calls per task versus Mimo's 2.63; Mimo batched changes and fixed them afterward.

The developer's qualitative observations surfaced two particularly frustrating behaviors. First, Terra demonstrated a "tendency to maximize context window, even when the things it read were evidently not relevant," acknowledging its own overreach but still consuming tokens. Second, Terra displayed what the developer called an "obsession with Git," constantly examining commits and diffs as a debugging tool so aggressively that the developer modified AGENTS.md instructions to limit the behavior—which Terra then largely ignored. These traits make Terra perform well on benchmarks (which reward thorough exploration and incremental verification) but feel "sluggish" in practice.

Ultimately, the developer concluded that while GPT-5.6-terra is "not a bad model," they would not use it at current pricing, primarily because of the volume cost and the longer time it takes to complete tasks. The 48.5% token premium, multiplied across many tasks, compounds into a significant economic and latency penalty.

The comparison reveals a tension between benchmark performance and practical efficiency in AI agents. GPT-5.6-terra's behavior — maximal context exploration, small incremental edit-and-check loops, and extensive tool use — are traits that often correlate with high benchmark scores, as the developer notes. However, these same patterns translate to inefficiency in real-world tasks where context window size directly impacts cost and latency. The specific inefficiencies (reading full files rather than targeted spans, running unfiltered commands, obsessive Git inspection) suggest that Terra may be trading practical awareness for exploratory thoroughness. The developer's decision to adjust instructions to limit Git usage, which Terra then ignored, hints at a model that is difficult to steer away from its learned behaviors even with explicit guidance. The 48.5% token inflation is not merely a curiosity—it compounds across many tasks, making the economic and time costs substantial enough to outweigh Terra's correctness on individual tasks.

No comments yet. Be the first to share your thoughts!