AWS push-based NIXL connector lifts vLLM TTFT 3x

Prefill-decode disaggregation in vLLM splits LLM inference into two phases that run on separate engine instances: prefill, which processes the input prompt and builds the key-value (KV) cache, and decode, which uses that cache to generate output one token at a time. This separation lets each phase scale independently, but it requires transferring the KV cache from the prefill instance to the decode instance before token generation can begin. The default pull-based NIXL connector performs that transfer by having the decode instance allocate destination blocks, prepare the transfer, and issue an NIXL READ after prefill completes. Those coordination and setup steps run serially after prefill and before decoding, extending time-to-first-token (TTFT).

The push-based connector changes the order of operations. The decode instance registers its target blocks at allocation time, while the prefill instance is still computing the KV cache. Once prefill finishes, a dedicated background thread on the prefill side matches the registration, prepares NIXL descriptors, and submits an NIXL WRITE directly into the decode instance's registered memory. This overlaps decode-side setup with prefill and moves transfer submission off the decode worker's main loop, so token generation is not stalled by per-request descriptor work. The body notes a tradeoff: pull mode lets the router pick a decode instance after prefill for load-aware routing, while push mode selects the decode instance earlier to enable that overlap.

The benchmark used a 1P1D deployment on Amazon EC2 P5en instances connected with EFA, running Qwen3-32B with prefix caching disabled to force full KV transfer per request. At low load and short prompts the two modes performed similarly, but as request rate and input length grew, pull mode's TTFT reached about 1,744 ms at 8 QPS and 16k input, while push mode stayed around 540 ms. TPOT showed the same pattern: pull climbed to about 10.4 ms at 4–8 QPS and 16k, while push held near 7 ms. The body states results depend on workload and configuration, and that AWS is exploring bringing the background threading model to pull mode via an open pull request.