What hardware did they test Magnitude on?

They tested it on an Apple M5 with 10 cores and 16GB of unified memory. The profile finished in under a minute.

Why is agent work different from chat for local models?

Agent trajectories accumulate for twenty or thirty turns, with tool outputs and file contents, making the context larger than the weights. Mistakes in tool calls have no partial credit, so high precision is essential.

What does the setup process involve?

Setup has three steps: choose a model, install it, and pick where to use it. The tool asks which harness you want to connect and writes that configuration itself.

Large Language ModelsOpen-Source AIAI Business & IndustryDaily Dose of Data ScienceSep 2, 2026Sep 3, 2026, 06:00 JST

Local AI agent slow? Magnitude auto-tunes your machine

Send on LINE

3 Key Points

What happened
Magnitude, an open-source inference server, profiles your hardware and recommends the best model configuration. In a test on an Apple M5 with 16GB, it suggested Gemma 4 E2B at 4-bit QAT, predicted at 43–51 tokens per second.
Why it matters
Agent work is slower and less forgiving than chat because it accumulates long conversation histories and needs high precision. Magnitude solves the "which model to run" problem that existing tools leave to users, saving time and avoiding mistakes.
What to watch
The tool measures memory bandwidth, which predicts generation speed, and it handles setup for harnesses like Pi and Codex. It also offers configurations for balanced, smartest, or fastest performance.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

Context & Analysis

The article argues that local model tools were designed for chat, not agent work. In agent use, the conversation history grows, memory bandwidth limits speed, and precision is crucial. Magnitude addresses this by profiling hardware.

It measures memory bandwidth, which predicts generation speed, and runs test inferences to account for real-world behavior. It then recommends complete configurations, including model, compression, context size, and expected speed range, sorted by balanced, smartest, or fastest.

The test on an Apple M5 showed Gemma 4 E2B at 4-bit QAT as the balanced pick, holding 50K context in 4.6GB, predicted at 43–51 tokens per second. The tool also handles concurrency and speculative decoding automatically. In a real test with wifi off, it correctly identified no PII in client files, confirming it ran entirely locally.

FAQ

What hardware did they test Magnitude on?: They tested it on an Apple M5 with 10 cores and 16GB of unified memory. The profile finished in under a minute.
Why is agent work different from chat for local models?: Agent trajectories accumulate for twenty or thirty turns, with tool outputs and file contents, making the context larger than the weights. Mistakes in tool calls have no partial credit, so high precision is essential.
What does the setup process involve?: Setup has three steps: choose a model, install it, and pick where to use it. The tool asks which harness you want to connect and writes that configuration itself.

Daily Dose of Data ScienceRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.