vLLM's chunked prefill stops long prompts stalling others

The article opens its explanation with the two phases every LLM request moves through. Prefill handles the whole input prompt in parallel and is compute-heavy, building the request's KV cache and producing the first output token. Decode then produces the rest one token at a time, repeatedly reading model weights and the KV cache, which makes it mainly a memory-bandwidth problem. That difference is what sets up the problem chunked prefill is meant to solve: a decoding request needs the GPU back between steps, while a large prefill pass wants to run uninterrupted.

The article's illustration is Request A already streaming tokens when Request B arrives with a 32K-token prompt. Without chunked prefill, B's prompt is processed in one large pass and A's next token has to wait, so A's response appears to freeze. Chunked prefill splits B's prompt into smaller token ranges, so vLLM can process one chunk, give active requests another chance to decode, and then continue B's prefill in later iterations — the same prompt is still processed, just not as one uninterrupted block of GPU time.

The stakes lie in the tradeoff the article lays out. Smaller chunks keep active responses moving and reduce inter-token latency spikes, but B cannot generate its first token until every chunk is processed, so larger chunks usually improve its time to first token. Chunks that are too small add overhead, since later chunks must attend to token states from earlier ones, repeatedly reading more of the KV cache and potentially reducing GPU utilization. Scheduling prefill chunks alongside active decodes may let both kinds of work share an iteration more efficiently, and in vLLM V1 the --max-num-batched-tokens setting governs how many tokens can be scheduled per iteration. Because there is no universal best chunk size, which direction teams lean is likely to hinge on whether their traffic is dominated by long prompts and strict streaming targets or by a priority on time to first token.