Uzu hits 92–117 tok/s on base M5, beating llama.cpp

Local LLM inference runs at batch size one: one app, one active sequence, one device. That removes cross-user batching, but also narrows the problem — the engine optimizes a single decode on known hardware. Uzu attacks two quantities at once: bytes transferred per pass and accepted tokens per pass.

The bytes side starts with a hard memory-bandwidth limit. A base M5 moves 153 GB per second, and a 4-bit Qwen3.5 9B checkpoint weighs about 5.2 GB, capping ordinary decoding at roughly 153 ÷ 5.2 ≈ 29 tokens per second. Uzu's Medium checkpoint uses 4-bit asymmetric integer quantization, and its Metal kernels run the speculative verification step on the M5's int8 matrix path. A block-diagonal Random Hadamard Transform of 32 values spreads outliers before quantization, matching the Apple GPU kernel's SIMD width.

The tokens side uses DFlash and Weaver to draft 16-token trees, while rollback-free verification handles Qwen3.5's Gated DeltaNet layers — 24 of its 32 — by computing candidate outputs without touching live state, then committing only along the accepted path. Mirai's published 27B benchmark reports 2.1× MTPLX, 3.8× llama.cpp and 4.4× MLX, with speculation lifting 35.54 to 114 tokens per second. Speed still depends on the workload: code and maths accept more drafts than open-ended conversation.