AWS cuts agent TTFT up to 3.3x with bi-directional KV transfer

The feature addresses a structural inefficiency in prefill-decode disaggregated serving. In the conventional one-directional flow, the decode node holds KV cache entries for both the prompt and the newly generated tokens, but the prefill node cannot retrieve them on the next turn. That forces the prefill node to recompute KV projections for previous response tokens, and it also increases the chance of cache eviction for active sessions on prefill nodes. When a full cache miss occurs, the prefill node must recompute the entire conversation history, including system instructions, prior user requests, tool results, and model-generated responses. AWS originally built and validated bi-directional KV transfer on Trainium instance clusters connected with EFA and contributed it to vLLM as an accelerator-agnostic capability. The benchmark results on an Amazon EC2 P5en GPU cluster with EFA indicate the benefits carry over to GPUs, according to the post.

Two controls shape when reuse actually helps. The decoder_kv_blocks_ttl setting keeps a conversation's KV blocks available for a fixed period, and the kv_recompute_threshold setting, defaulting to 64 tokens, decides whether prefill pulls remote KV or recomputes locally. The post notes that for a short reusable prefix, interconnect and coordination overhead can exceed the time needed to recompute the tokens. It also points to extending a heartbeat-renewed lease to the decode side as a natural next improvement, so blocks could be reclaimed as soon as a conversation goes idle rather than at the end of a fixed timer.

The benchmark deliberately disabled prefix caching on the prefill node to emulate cache eviction or a router selecting a different prefill node without the conversation's KV blocks. Under that condition, with the feature off, TTFT rose steeply as context grew: Qwen3-32B climbed from 158 ms at 2k to 778 ms at 24k, and GLM-5-FP8 from 500 ms at 2k to 2,334 ms at 20k. With it on, Qwen3-32B rose only from 122 ms to 460 ms, and GLM-5-FP8 from 358 ms to 702 ms. The post attributes the larger gap for GLM-5-FP8 to its higher per-token recompute time, which makes KV reuse pay off more as models and contexts scale.