
What happened
llama.cpp hit version 1.0 with --moe-cache-cpu-share, an option that hands 0–1 share of expert-cache misses to the CPU instead of moving them over PCIe to the GPU.
Why it matters
For large models where expert data does not fit in memory, letting the CPU process a share of misses keeps the GPU from stalling, so massive models run more efficiently—smoother generation, at the cost of slower prompt processing in some cases.
Summaries like this, every morning.
llama.cpp's move to version 1.0 brings a shift from a simple hit-rate expert cache to an LRU (Least Recently Used) cache, which prioritizes experts used in the most recent batch and discards old ones even if their hit rate was high. Used experts vary with context—coding, thinking, prose—so a cache that follows recent usage is more effective than one that just counts hits.
The --moe-cache-cpu-share option itself is not a novel technique; the popular inference engine Strata actively uses a similar approach, tuning strongly to each model. llama.cpp's difference is its generality: it can run any model with a GGUF file, whereas Strata aims for high performance on specific models. With tuning, llama.cpp can reach a decent level for large MoE models.
A walkthrough on a GeForce RTX 3090 with DeepSeek V4.1 Flash 1017 shows the practical trade-off: text generation improved as the CPU share rose to 0.9, but prompt processing slowed, likely because CPU involvement makes the GPU wait. The lack of an auto setting—the patch was not committed—means users must manually balance these speeds. For models where experts fit entirely in physical memory, the option is unnecessary and often harmful, as transferring experts over PCIe to the GPU is faster than CPU processing.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
Reson8 released Rumble-2-Flash, a 500 million-parameter transcription model supporting over 40 languages, as a…
At Microsoft Ignite 2026, Microsoft said Microsoft Perception is now generally available, combining red, blue…

At Microsoft Ignite 2026, Microsoft released a public preview of Windows Server 2028, which adds AI inference…

AUMOVIO chose Databricks Unity Gateway as its single route to LLMs after a cross-functional taskforce picked o…

CADMAGEDDON scores 0.991 on BenchCAD's 64³ voxel IoU over a 1000-part subset, above vendor-reported 0.963, 0.9…

MIT Technology Review's AI Hype Index describes AI spreading everywhere, citing Google's AI Overviews costing…
