DeepSeek open-sources DeepInfer for 26x faster cache hits

DeepSeek's move to open-source DeepInfer reveals how the company turned an architectural choice into a serving advantage. The V2 paper's Multi-head Latent Attention cut the KV cache by 93.3% compared to DeepSeek's earlier 67B model, shrinking the data that must be moved on every cache hit. That reduction is what made disk storage viable in the first place, since reading a full per-head KV cache from disk could take longer than recomputing it on the GPU.

The second ingredient is 3FS, DeepSeek's own distributed file system, which splits files into chunks across thousands of SSDs so a read pulls from many drives in parallel. When a request arrives, the system matches its prefix against cached blocks and fetches them into host memory, then copies them to the GPU layer by layer while the model is already running. Because every prefill machine can reach every cached prefix, a request does not have to return to the machine that served it last. Disk is also cheap, so cached prefixes can stay for days, which the DeepInfer docs put at 3 days for agent traffic and 7 days for chat.

The pricing effect is visible on deepseek-flash, where a cache hit costs 50x less than a miss today. For API users, that creates a practical incentive to order prompts so the stable parts, such as system prompts, tools, and documents, sit at the front, since only an exact match from the very first token counts as a hit.