Memory-Saving Tech for LLM Serving Explained

The article addresses a core serving problem: while model weights stay fixed, the KV cache grows with every token and every active sequence. It points out two costs: memory usage and repeated reads during generation. The techniques are grouped by what they reduce, such as GQA for heads or sliding windows for tokens.

A key theme is that many optimizations, like CLA and MLA, are not generic engineering tweaks but change the model's architecture and training. For example, CLA requires the checkpoint and engine to agree on cache ownership, and MLA's projection weights are part of the trained model. This means adoption requires planning from the model design stage, not just at serving time.

The choice of technique likely involves a trade-off between memory savings and model quality. Methods like GQA and sliding windows can alter what the model learns or sees, potentially affecting accuracy. The outcome of applying them probably depends on whether the quality loss is acceptable for the memory gain, and whether they can be combined without compounding degradation.