Seven MoE routing strategies — including DeepSeek-V3's node-limited routing — explained for AI engineers

The issue frames routing as the place where MoE's promised compute savings can quietly disappear. Sending every token to its highest-scoring experts looks efficient, but if many tokens prefer the same experts, those experts become overloaded while others sit idle; with fixed capacity and a dropping rule, some assignments may be dropped, and uneven workloads plus communication overhead across GPUs can erode the savings.

The seven designs each handle that tradeoff differently. Expert Choice reverses the selection so each expert picks its highest-scoring tokens within a fixed capacity, but equal expert workloads can still mean unequal computation per token. Dynamic routing lets the number of active experts vary, and global matching such as BASE Layers accepts that some tokens may miss their first-choice expert to balance the batch. Hashing removes the learned gate entirely at the cost of flexibility, Soft MoE has experts process weighted mixtures of token representations, and hardware-aligned grouping — as in DeepSeek-V3's node-limited routing — narrows where a token can travel. The newsletter cautions that these are routing designs, not interchangeable serving-engine settings, and that some come from training or vision research and need care for autoregressive generation.

The practical thread is measurement. Evenly distributing tokens does not guarantee low latency, because expert batches may still be small and moving tokens between devices takes time. Profiling should separate routing, token movement, expert computation, and combination so the slowest stage is visible, and the issue points to a 10-week inference-engineering roadmap on GitHub for building and benchmarking that path.