Apple's MLX runs Muse Spark Open at over 36 tokens per second on four Mac Studios

Apple's MLX is an open-source array framework that leverages the unified memory architecture of Apple silicon, letting researchers run inference and fine-tuning methods on their own hardware. In this post, Apple shows how the latest version of MLX, released today, runs the frontier-class model Muse Spark Open — with nearly a trillion parameters — on a cluster of four Mac Studios, generating over 36 tokens per second.\n\nThe post benchmarks three frontier-class models on Mac Studio: Muse Spark Open (816B), Mistral Medium 3.5 (128B), and Qwen 3.8 (27B). Muse Spark Open is a mixture of experts model, which keeps only a fraction of its experts active, reducing compute at inference, and it is evaluated in NVFP4 precision. The smaller Qwen and Mistral models fit comfortably on a single Mac Studio with 512GB of RAM, but Muse Spark Open would require more than 1.6TB of RAM in its native precision for the model parameters alone, so it requires parallelization across multiple Mac Studio computers.\n\nApple describes two approaches to splitting large models: pipeline parallelization, which splits a model across its depth, and tensor parallelization, which splits it across its width. These rely on fast inter-machine communication, and macOS 26.2 introduced a low-latency RDMA driver over Thunderbolt 5 that MLX builds on, providing new communication collectives called jaccl. Apple also notes that the Mac Studio is energy-efficient for LLM inference, with peak power consumption of about 200W for the M3 Ultra and 300W for the M5 Ultra — a fraction of the power used by a common toaster.