Apple researchers introduce SpecMD framework and Least-Stale eviction policy to optimize expert caching in Mixture-of-Experts models

Apple researchers developed SpecMD, a standardized framework for benchmarking cache policies on Mixture-of-Experts (MoE) models—neural networks that activate only a subset of parameters during inference—across various hardware configurations.

The team proposed Least-Stale, a novel eviction policy that exploits MoE's predictable expert access patterns, reducing collision misses by up to 85× over LRU (Least Recently Used, a standard caching method) and achieving over 88% hit rates with up to 34.7% Time-to-first-token reduction on OLMoE at only 5% or 0.6GB of VRAM cache capacity.

Benchmarking revealed that MoE expert access does not follow temporal locality assumptions typical of existing cache strategies like LRU and LFU (Least Frequently Used).