AWS opens MoE kernel fixes for 10% higher throughput on SageMaker AI

Fused MoE merges token routing, data shuffling, and expert network calculations into a single batched GPU operation instead of running them as separate sequential steps. That structure helps latency but makes the kernel harder to tune, and it is especially sensitive to the constraints of Nvidia's Hopper architecture — the generation that includes the H200 GPU.

The first bottleneck shows up during decoding, when the system handles one token per user sequence at a time. The batch size shrinks to a single row or fewer than 64 rows, while the expert weight matrices stay large. Hopper's WGMMA instruction expects a fixed 64-row layout for its first operand, so a thin token vector leaves much of the Tensor Core capacity unused. AWS's Swap-AB transposition promotes the large expert dimension into that leading slot, which saturates the layout and lets Hopper prefetch memory asynchronously.

A second gain came from routing FP8-quantized MoE work to vLLM's Triton backend rather than the default FLASHINFER_CUTLASS path. Triton fuses scaling, matrix multiplication, and expert aggregation into one pass through fast on-chip memory, which suits the small, fragmented token batches of live serving. A third fix removed repeated Triton JIT recompilations triggered by changing token counts, which matters in multi-GPU clusters where all eight GPUs must stay in step. Together these changes delivered up to 10% higher throughput and up to 9% faster Time to First Token on H200 GPUs.