ax-engine hits 76 tok/s on M5 Max, beats mlx-lm

The project starts from a hardware quirk of Apple Silicon Macs: the GPU and CPU share the same Unified Memory Architecture. That means running a local LLM is often limited not by raw compute but by memory bandwidth, the rate at which the model's weights and KV cache can be read. In practice, a model may load fine yet generate tokens more slowly than expected.

ax-engine targets that gap by focusing on execution and serving rather than a chat UI or a cloud service. Its design bets include MTP decoding, in which an auxiliary prediction head proposes several tokens ahead and the main model verifies them, plus Prefix reuse and the ability to serve multiple models from a single process. The developer stresses that MTP's benefit is not constant — it shifts with the model architecture, whether MTP is supported, the quantization format, prompt and context length, the number of tokens generated, the Apple Silicon generation, and Unified Memory capacity. That is why the guidance is to compare like with like: same model, same quantization, same prompt.

The benchmark table itself is presented with placeholder values in this write-up, so the reproducible public figure is the reported M5 Max result of over 76 tok/s and the roughly 2.4 to 2.7x edge over mlx-lm under some conditions. The developer is explicit that these are not fixed numbers and can change widely by configuration. To widen the picture, they are asking for results from M1 through M5 machines with 16 GB up to 64 GB or more of memory, across quantization formats, along with chip and memory size, macOS and ax-engine versions, model name, context length, decode and prefill speeds, memory use, and any errors. Known limitations include uneven model support, MTP's dependence on model and settings, install and model-conversion issues, and no guarantee of compatibility with every OpenAI-compatible client.