Evolving Infra cuts agentic LLM P95 TTFT up to 14× vs Nvidia Dynamo

The release targets a specific mismatch in how agentic AI workloads behave. Agents generate long, irregular context sequences and their balance between prefill (reading the prompt) and decode (generating output) shifts constantly. Existing disaggregated serving systems, Dynamo included, set their compute allocation from offline profiles that go stale as those patterns change, which is where the latency penalty shows up at high concurrency.

Evolving Infra's answer is to split the problem in two: an online planner adjusts prefill-decode splits on the fly, while an agent-aware distributed KV manager handles cache eviction and transfer across nodes. That design is what produced the up-to-14× P95 time-to-first-token and up-to-4.1× mean end-to-end latency improvements over Dynamo in the 256-GPU evaluation on GLM-5.2 and Kimi K2.6.

The author list—Song Han of MIT's Han Lab, Ion Stoica of UC Berkeley, Tushar Krishna of Georgia Tech, and Huizi Mao among others—puts the work at the intersection of systems research and production serving. The open-source release means infrastructure teams can evaluate these ideas directly rather than waiting for a vendor to absorb them into a commercial stack.