Ai2's Olmo-core 3 hits 52,000 tokens per second, topping Megatron-core

Mixture-of-experts models work differently from dense ones: they split computation across specialist portions of the model each time a token is generated, while dense models activate the whole model. That lets a MoE hold far more total parameters while computing with only a small number of them, cutting the compute per token. The catch, as Ai2 describes it, is that the full model still has to be stored across GPU memory, and coordinating networking between experts during training creates extra costs.

Olmo-core 3 is Ai2's answer to that catch. It uses expert parallelism to spread experts across multiple GPUs, so each card stores only part of the full expert pool. It also splits the model's layers across groups of GPUs and uses a distributed optimizer that spreads optimizer state across GPUs instead of keeping full copies on each one. Ai2 says this reduces memory overhead because the entire model and its training state need not be held in memory at once, and it adds support for MXFP8, a number format that represents some values with fewer bits to cut computation and data movement.

The stated ambition is about access: trillion-parameter models are often beyond the reach of those without state or enterprise infrastructure, and Ai2 says the framework would also let researchers adapt MoE training to different hardware and experiment with routing and parallelism. What the outcome hinges on is whether those gains hold outside Ai2's own benchmarks — especially given the comparison here is against Nvidia's Megatron-core on Nvidia's own GPUs. For researchers watching the space, the open GitHub release is the practical test.