
What happened
In test E13, single-session retry used 4 model calls and 13,822 input tokens; parallel branches used 2 calls and 4,537 tokens, and both passed all 20 tests.
Why it matters
The result suggests that retrying repeatedly in one session inflates token use much faster than generating independent branches, which matters for teams managing AI agent costs.
Summaries like this, every morning.
The experiment was designed around a realistic legacy codebase: three payment ledger modules (ingest.py, reconciler.py, summary.py) seeded with multiple bugs and an O(n^2) double-loop performance bottleneck. Mode A simulated the common developer habit of patching the same session over and over, while Mode B generated two clean-context branches from scratch — one conservative patch and one clean O(n) redesign — and let a deterministic judge pick the winner. The trace data shows that Mode A's input tokens grew from 2,377 in Turn 1 to 4,383 in Turn 4, a 1.84x increase, which the report identifies as the mechanism behind the token gap.
The judge's decision proved consequential. The conservative branch passed all functional tests but retained the double loop, scoring 80/100 and running the 2,000-event benchmark in 202.952 ms. The clean redesign scored 100/100 and ran the same benchmark in 0.962 ms. Without a deterministic judge weighing structural checks and benchmarks, the report argues, an LLM reviewer might have favored the smaller-looking conservative patch. The practical recommendation is to switch to parallel branches after two failed retries, before sunk-cost bias and context bloat take hold.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.