LLM serving: why continuous batching wins

The article explains that a GPU's forward pass is bottlenecked on reading weights from memory, not on math. This is why batching helps: loading weights once and pushing many sequences through the same pass spreads the memory cost. However, LLMs are autoregressive—each pass produces one token and depends on previous ones—so a fixed batch runs at the pace of its slowest member.

Static batching works for fixed-output models like classifiers or embedding models, where every request finishes at nearly the same step. Dynamic batching adds a timer but still treats the batch as one unit, so fast requests wait for slow ones. Continuous batching, with iteration-level scheduling, is the only strategy that keeps the GPU saturated when output lengths vary.

The article also highlights chunked prefill as a technique to mitigate stutter when a long prompt joins the batch. It splits prompts into smaller chunks, giving the scheduler more chances to run decodes. There is no universal best chunk size—it depends on the model, GPU, prompt distribution, and the latency metric an SLO targets. For live traffic with variable output lengths, continuous batching is the only strategy that holds up, and every major engine enables it by default.