Explainer: KV キャッシュ cuts recomputation but eats memory

The piece is written for readers who keep meeting terms like "KV cache" and "prefill / decode" in LLM news and want to hold the machine in their heads once with small numbers. It builds the picture from the bottom: each token becomes a list of numbers (an embedding), Attention mixes in information from related words, and an FFN processes each word separately, with those two blocks stacked 32 to 100 times. Query, Key and Value are introduced as a library search — Query is the search word, Key is each book's title, Value is the book's contents — and the same single example sentence runs through the whole calculation.

The memory cost gets a concrete shape. The explainer gives the KV cache size as 2 (K and V) × layers × KV heads × head dimension × bytes per element × tokens × simultaneous requests, and works it out for Llama 3.1 8B (32 layers, 8 KV heads, dimension 128, up to 128K tokens) in FP16: 128 KiB per token, 1 GiB at 8K tokens, 16 GiB at 128K tokens, which it notes is about the same as the FP16 model itself, and ten times that with 10 simultaneous requests.

It then maps the fixes that show up in news coverage back to that formula — GQA / MQA (sharing K and V across Query heads), quantizing the cache from FP16 to FP8 / INT4, vLLM's PagedAttention, sliding windows that discard old tokens, and DeepSeek's MLA, which compresses K and V. Prompt caching in the Claude and OpenAI APIs is described as reusing this same cache across requests, which is why an identical prefix at the start is what makes reuse possible.