How to Route AI Calls Across Models Without Losing Cache Savings

The article addresses a practical problem that looks simple on the surface but has a hidden implementation cost. When a business or developer adds a router to send easy calls to a cheaper model, the expectation is immediate savings. However, how language model API calls work in practice—especially how prompt caching works—defeats naive routing. Each call in a task carries the full accumulated context, and the cache that makes re-processing cheap is model-specific. Switching models discards it.

The solution is not smarter routing logic but a shift in routing granularity: instead of routing per call, route per task. This requires tracking session state and pinning the model choice across multiple API calls. The article demonstrates that this layer can be implemented as an open-source proxy (Plano) that sits between the agent and the providers, so teams do not have to rewrite their agent code. The 2x usage reduction reported on a real agent suggests this is not a marginal optimization but a structural necessity for multi-model routing in production.

The architecture also highlights why the routing model itself must be tiny (described as 1.5B parameters)—a routing decision that costs as much as the call it routes saves nothing. By using human preference data (what developers actually pick) instead of benchmark rankings, the router reflects real-world trade-offs between cost and quality, and since the policy is defined in plain YAML config, the routing strategy can adapt when provider pricing changes without touching agent code.