Induction AI busts benchmark cache with trial-id tool

Induction AI builds context optimization for AI agents, and it runs benchmarks both to test new optimization ideas and to verify cost savings without compromising accuracy. In early experiments, benchmark trials shared a prompt cache, which made later trials cheaper and skewed cost measurements. The company typically runs multiple trials of the same task and computes statistics on cost and accuracy distributions, but the prompt cache has no idea which conversation a request belongs to: any request whose opening tokens match a recently processed prefix reads that prefix at a discount.

Because trials of the same or related tasks start with the same tools, system prompt, task message, and often the same first few tool calls, trial 2 could read trial 1's opening turns from the cache. That includes trials launched in parallel, since the cache entry exists as soon as the first request is processed. Induction AI's cache invalidation simulator showed that tools sit at the very top of the prefix and that adding a tool leaves nothing cached — which pointed to the fix: give each trial a tool whose name carries a trial id, different across trials so no trial matches another's prefix from its first tokens, but the same for the whole trial so its own turns still cache against each other.

The team uses benchmarks such as tau3 and Toolathlon to measure how much context optimization reduces inference costs while maintaining task success rates. It notes that injecting a unique dummy tool turned out to be a simple solution, and a reminder that the infrastructure surrounding an LLM can influence benchmark results as much as the model itself.