Redis LangCache: 15x faster, 70% cheaper on cache hits

The article, published on the Daily Dose of Data Science blog, pairs two explanations: a practical caching layer for LLM apps and a deeper walkthrough of how mixture-of-experts models are served. On the caching side, the pitch is that production assistants often receive differently worded versions of the same question. Prefix caching cannot help because it only reuses computation on an exact match, and even then a response is generated again. LangCache instead stores complete question-response pairs outside the model, embeds the incoming question, searches stored responses, and applies a configured similarity threshold. A hit returns the earlier answer, while a miss falls back to the model and can store the new response for later.

On the serving side, the article follows a token through an MoE layer. A router scores experts and selects a subset per token; the runtime dispatch groups those assignments into expert matrices, and combine returns results to their original tokens. It stresses that activated parameters estimate computation for one token, while resident parameters determine weight memory — the model card's 3.3 billion activated parameters do not describe the deployment's memory footprint.

The piece also draws a line between optimizations that change only execution, such as fusion, grouped GEMM, and placement, and those that change numerical values or which computation runs, such as quantization and reducing top-k. A cited paper, Training-Free Halving of Activated Experts, reports that standard reduction from eight experts to four on Qwen3.6-35B-A3B lost 4.65 MMLU points, while an adjusted reference mass cut the measured loss to 0.35 points. Whether LangCache's stated savings materialize in any given deployment appears to hinge on how much safe repetition that workload actually contains.