AIToday
Large Language ModelsAI Coding AssistantsOpen-Source AIZenn AI/MLPublished: Nov 18, 2026, 16:00 JST

llama.cpp 1.0 adds --moe-cache-cpu-share for big models

Send on LINE
llama.cpp 1.0 adds --moe-cache-cpu-share for big models

3 Key Points

  1. What happened

    llama.cpp hit version 1.0 with --moe-cache-cpu-share, an option that hands 0–1 share of expert-cache misses to the CPU instead of moving them over PCIe to the GPU.

  2. Why it matters

    For large models where expert data does not fit in memory, letting the CPU process a share of misses keeps the GPU from stalling, so massive models run more efficiently—smoother generation, at the cost of slower prompt processing in some cases.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

llama.cpp's move to version 1.0 brings a shift from a simple hit-rate expert cache to an LRU (Least Recently Used) cache, which prioritizes experts used in the most recent batch and discards old ones even if their hit rate was high. Used experts vary with context—coding, thinking, prose—so a cache that follows recent usage is more effective than one that just counts hits.

The --moe-cache-cpu-share option itself is not a novel technique; the popular inference engine Strata actively uses a similar approach, tuning strongly to each model. llama.cpp's difference is its generality: it can run any model with a GGUF file, whereas Strata aims for high performance on specific models. With tuning, llama.cpp can reach a decent level for large MoE models.

A walkthrough on a GeForce RTX 3090 with DeepSeek V4.1 Flash 1017 shows the practical trade-off: text generation improved as the CPU share rose to 0.9, but prompt processing slowed, likely because CPU involvement makes the GPU wait. The lack of an auto setting—the patch was not committed—means users must manually balance these speeds. For models where experts fit entirely in physical memory, the option is unnecessary and often harmful, as transferring experts over PCIe to the GPU is faster than CPU processing.

FAQ
What does the --moe-cache-cpu-share parameter actually do?
It takes a value from 0 to 1. At 0 (default), expert misses are not processed on the CPU; at 1, every miss is computed on the CPU; intermediate values give that share to the CPU.
Is it always beneficial?
No. If all experts fit in physical memory, --moe-cache-cpu-share is ineffective and often counterproductive. It is worth trying mainly when GPU performance is modest or VRAM is small (e.g., 8GB).
Can I tune the share automatically?
Not currently. The feature has no auto setting—an earlier patch for auto mode was passed over—so users must adjust the value manually. In testing, 0.9 gave the best text-generation speed, but prompt processing got slower.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleQuang Luong uses OpenAI's Codex to find Java4Shell in one hour