Magnitude is an open-source inference server that profiles your machine and recommends models.
It measures memory bandwidth and runs test inferences to pick configurations.
In testing on an Apple M5 with 16GB, it recommended Gemma 4 E2B at 4-bit.
What happened
Magnitude, an open-source inference server, profiles your hardware and recommends the best model configuration. In a test on an Apple M5 with 16GB, it suggested Gemma 4 E2B at 4-bit QAT, predicted at 43–51 tokens per second.
Why it matters
Agent work is slower and less forgiving than chat because it accumulates long conversation histories and needs high precision. Magnitude solves the "which model to run" problem that existing tools leave to users, saving time and avoiding mistakes.
What to watch
The tool measures memory bandwidth, which predicts generation speed, and it handles setup for harnesses like Pi and Codex. It also offers configurations for balanced, smartest, or fastest performance.
Ask the AI about this article →
The article argues that local model tools were designed for chat, not agent work. In agent use, the conversation history grows, memory bandwidth limits speed, and precision is crucial. Magnitude addresses this by profiling hardware.
It measures memory bandwidth, which predicts generation speed, and runs test inferences to account for real-world behavior. It then recommends complete configurations, including model, compression, context size, and expected speed range, sorted by balanced, smartest, or fastest.
The test on an Apple M5 showed Gemma 4 E2B at 4-bit QAT as the balanced pick, holding 50K context in 4.6GB, predicted at 43–51 tokens per second. The tool also handles concurrency and speculative decoding automatically. In a real test with wifi off, it correctly identified no PII in client files, confirming it ran entirely locally.
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
Vertiv Holdings Co. (NYSE: VRT) announced its wholly-owned subsidiary has entered into an agreement to acquire…
Interactive Brokers has begun connecting its platform with AI tools including ChatGPT, Claude, and Grok, and o…
The article highlights three AI healthcare stocks from a screener: Pfizer (PFE), Tempus AI (TEM), and Medtroni…
MBody AI Ltd. (NASDAQ: MBAI) announced that its MBody AI Orchestrator platform has been shortlisted for AI Dep…
A new report by Alipay+ and S&P Global, based on a survey of 6,000 consumers across nine markets in Asia, Euro…
MaxLinear's Infrastructure business is projected to grow, driven by demand from AI data centres
