KV キャッシュ cuts AI answer wait from 452秒 to 14秒

A small transformer with random weights, run on a single CPU thread, and a queue simulation that replays 1,000 requests arriving at once. That setup is deliberately modest, and the author notes the timings apply only to this implementation, not to any production system. Within that scope, the split between prefill (processing the prompt once) and decode (producing one new token at a time) is what makes caching worthwhile: the keys and values for earlier tokens never change, so recomputing them is pure waste.

The queue simulation adds a second layer. With caching alone, median time to first token fell from 452 seconds to 24 seconds; adding scheduling that lets finished requests leave the batch brought it to 14 seconds. The largest single effect was caching, and the final gap mixes two changes. It also points to the memory cost: each active request holds its own cache, and a 4,096-token conversation at the assumed dimensions takes 512 MiB, which caps how many requests can run at once.

The practical takeaway the author offers is narrow. If you run a model on your own machine for yourself, caching is already doing the work; batching matters once a server is shared. The figures are a demonstration of the pattern, not a benchmark of any company's system.