What is the KV cache and why does it matter?

The KV cache stores key and value vectors for every token the model has already seen, so during decode (output generation) the model can look them up instead of recomputing the entire conversation history. It is stored in GPU memory separately from the 8.6 GB weights and grows as the conversation grows; on Qwen3.5-4B, a cached token is around 130 KB.

Why can the server serve multiple requests at once?

The server batches requests together: each forward pass streams the weight matrices once and multiplies them against stacked input tokens from all active requests simultaneously. Because the weight-stream is the expensive step and the arithmetic is cheap, ten requests get their next token for the cost of one weight-stream.

Large Language ModelsHacker NewsAug 23, 2026Aug 23, 2026, 10:00 JST2 min read

LLM Serving Primer: How Inference Handles Requests

Send on LINE

Key takeaway

A language model server works by streaming fixed weight matrices through fast compute cores once per output token.
Multiple requests batch together to share each weight-stream, while a KV cache stores previously computed token data so decode does not reread history.
This primer explains the machine before the series demonstrates what breaks when you push each part.

3 Key Points

What happened
Manas Pathak published a technical primer explaining how language model servers process requests, covering the Qwen3.5-4B model (8.6 GB of weights) and the two-phase request cycle: prefill (reading the prompt in one pass) and decode (generating output token-by-token).
Why it matters
Understanding this serving architecture is foundational for the five-part series that follows, each of which isolates a failure point by turning one operational knob. The primer reveals the core constraint: weights must stream from GPU memory to compute cores for every token, making weight-streaming—not arithmetic—the bottleneck.
What to watch
The series will examine three levers (max_num_seqs, chunked prefill, and quantization) by breaking each one; Part 1 will explore why an 8.6 GB model serves only 7 requests per second, revealing how batching and KV caching interact under load.

Ask the AI about this article →

Summaries like this, in your inbox every morning.

Get news like this every morning — free

Context & Analysis

The article is a primer, not a report of an event—it is foundational scaffolding for a five-part technical series that will expose breakpoints in LLM inference. Pathak's framing rests on a single mechanical insight: the GPU's architecture splits storage from computation, forcing weights (8.6 GB) to live in slow memory while fast cores compete to use them. This split makes weight-streaming, not arithmetic, the dominant cost of inference. The primer isolates three structural concepts—the two-phase request cycle (prefill and decode), the KV cache as a memory optimization, and batching as a throughput multiplier—and then names three tuning levers (max_num_seqs, chunked prefill, quantization) that the series will test by breaking them one at a time. By establishing this shared model first, Pathak ensures each subsequent post can focus on a single failure mode without repeating the background.

FAQ

What is the KV cache and why does it matter?: The KV cache stores key and value vectors for every token the model has already seen, so during decode (output generation) the model can look them up instead of recomputing the entire conversation history. It is stored in GPU memory separately from the 8.6 GB weights and grows as the conversation grows; on Qwen3.5-4B, a cached token is around 130 KB.
Why can the server serve multiple requests at once?: The server batches requests together: each forward pass streams the weight matrices once and multiplies them against stacked input tokens from all active requests simultaneously. Because the weight-stream is the expensive step and the arithmetic is cheap, ten requests get their next token for the cost of one weight-stream.

Hacker NewsRead Original Article

Send on LINE

Get the latest Large Language Models news every morning

For example, today's edition would include:

Visko raises $10M, launches live AI video model OrbisSiliconANGLE AI · 1h ago
Runway unveils Solaris, an AI that generates app interfaces in real timeTHE DECODER · 1h ago
Google AI Search flags Facebook users as dangerTHE DECODER · 1h ago

AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.

Subscribe free →

Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →