Routing AI calls to cheaper models often fails because switching models mid-task destroys cached context, forcing expensive full re-processing. The solution is model affinity: pin a task to one model on first call, keep all subsequent calls in that task on the same model so the cache stays warm, and reset routing only when a new task begins. A team applying this four-stage routing pipeline (guardrail filter, router model, cost policy, and affinity) saw usage drop by 2x without changing agent code.
Summaries like this, in your inbox every morning.
Sign up free →What happened
An article explains why simply routing easy prompts to cheaper AI models often fails to save money—because switching models mid-task destroys the prompt cache (which costs 90% less on repeat access), forcing the context to be re-billed at full rates. Production routers solve this by pinning a task to a single model once chosen, keeping the cache warm across all calls within that task, and switching models only when a new task begins.
Why it matters
For teams running AI agents that make multiple sequential calls (planning, tool use, result analysis), naive routing can leave costs unchanged despite switching to cheaper models. Understanding how to preserve cache affinity means the cost savings actually materialize—the article reports that applying this exact four-stage pipeline (guardrail filter, router model, cost policy, model affinity) reduced usage by 2x on a real agent without changing a line of code.
What to watch
The pipeline is implemented in Plano, an open-source proxy that runs between your agent and model providers. The routing model uses Arch-Router, a 1.5B model trained on human preference data. Plano includes an observability console that shows per-request cost and which model answered each request, letting teams see routing decisions in real time.
The article addresses a practical problem that looks simple on the surface but has a hidden implementation cost. When a business or developer adds a router to send easy calls to a cheaper model, the expectation is immediate savings. However, how language model API calls work in practice—especially how prompt caching works—defeats naive routing. Each call in a task carries the full accumulated context, and the cache that makes re-processing cheap is model-specific. Switching models discards it.
The solution is not smarter routing logic but a shift in routing granularity: instead of routing per call, route per task. This requires tracking session state and pinning the model choice across multiple API calls. The article demonstrates that this layer can be implemented as an open-source proxy (Plano) that sits between the agent and the providers, so teams do not have to rewrite their agent code. The 2x usage reduction reported on a real agent suggests this is not a marginal optimization but a structural necessity for multi-model routing in production.
The architecture also highlights why the routing model itself must be tiny (described as 1.5B parameters)—a routing decision that costs as much as the call it routes saves nothing. By using human preference data (what developers actually pick) instead of benchmark rankings, the router reflects real-world trade-offs between cost and quality, and since the policy is defined in plain YAML config, the routing strategy can adapt when provider pricing changes without touching agent code.
No comments yet. Be the first to share your thoughts!
Log in to join the discussion
Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.
Get Started FreeFree · takes 30 seconds · unsubscribe anytime
1 minute a day. The AI essentials.
200+ sources · Email / LINE / Slack
