Magnitude inference engine beats llama.cpp by up to 2x

The Magnitude team's starting point is a mismatch between how inference engines are usually built and how local AI agents actually run. Datacenter engines are optimized for batching large numbers of requests, while projects like llama.cpp prioritize covering as much hardware as possible. Neither, in the team's view, fits a PC running an agent for a long time — a workload that involves repeated file reads, command execution, and often several agents at once.

Their answer is to compile the small computational units called kernels on the machine that will run the model, and to tune parameters there too. The reasoning is that even within the same Apple silicon or NVIDIA family, performance and memory layouts differ by product, so device-specific tuning can beat a generic kernel built for wide compatibility. The team's benchmark used a 4bit-quantized Qwen 3.6 35B A3B at a 64K token context, comparing against llama.cpp across both a Mac and an NVIDIA DGX Spark.

Beyond raw speed, the release addresses long-running and parallel use. Magnitude allocates only the memory needed to hold the model's weights, grows that region as a session lengthens, and frees it when an agent stops, while a shared prefix cache reuses computation for system prompts common across sessions. The team has also flagged what comes next: Expert streaming, a compiler that decides kernel configuration automatically, and optimization across CPU, GPU, RAM, and storage together. Whether the early gains hold as those features land is the open question.