Local 122B model scores 89% of Claude at 787× lower cost

The author runs Jarvis, a personal AI agent built on LangGraph, wired into roughly 90 tools—email (Outlook), calendar, notes, files, Office, WhatsApp, Discord, image generation, and sub-agents for longer jobs. It had always run on Claude. A month earlier, the author had tested five local models on a small benchmark (17 tasks, two agent frameworks, 12 coding and 5 general tasks) and found qwen3-coder:30b winning cleanly. On the smaller stage, the benchmark result held. But when the same 30B model was dropped into Jarvis's real environment—messier, broader, deeper in tool chaining—it crashed. The author measured that failure by replaying 28 real historical task prompts from Jarvis's own tracing logs (Langfuse), stratified across seven categories (calendar, code, email, files, general, messaging, notes). Claude's baseline was frozen: the actual production answer already logged, not re-run. Only the local model ran fresh, in a sandboxed harness that replayed real tool data from the original traces instead of stubbing generic placeholders.

Scoring was independent, using Claude Opus 4.8 as an LLM judge on a 1–5 scale mapped to 0–100. The author flags a real limitation: the judge is a Claude model, which introduces documented self-preference bias. That caveat mattered more in round 2, where the local model scored close enough for the bias to potentially shift perception. The process itself had three major bugfixes: a silent JSON parser failure in scoring, a mock harness returning canned text instead of real data (flattening the local model's score unfairly), and a Claude API rate-limit ghost-scoring some judge calls. The author verified everything, including eight round-2 tasks that scored exactly 70.0 (the neutral-failure sentinel) by hand-checking the judge's rationales—all eight were genuine "3/5" judgements.

Round 1: qwen3-coder:30b on a single RTX 3090 with a hard 16,384-token context limit (the shared GPU couldn't spare more VRAM alongside 18 GB of model weights). Claude averaged 89.4/100. qwen averaged 22.8/100 and never won a single category. The failure modes were stark: malformed tool-call tags (<function=send_email>...</function>) leaked into 7 of 27 final answers—raw syntax a user would have read, not a proper tool call. Tool-overlap recall (did the model call the tools that historically solved the task?) averaged 14.8% across the 18 tasks where tools were actually used. Two tasks orchestration-failed entirely—pilot-17 (email, 24 tool calls) and pilot-27 (messaging, 27 tool calls)—looping on the same already-answered tool instead of stopping. One shared failure mode: both models mishandled the mocked environment (where tool calls are intercepted). Claude over-claimed success (told the user an email was sent when it had been intercepted). qwen correctly reported that the send didn't go through—but it also leaked malformed syntax a quarter of the time and looped twice. Both models lied, in opposite directions.

Then the author upgraded the hardware: three RTX 3090s (72 GB VRAM), allowing a 122-billion-parameter mixture-of-experts model (Q3_K_M quantization, ~53 GB weights, no CPU spill) and a 256,000-token context window—a 16× increase. Same 27 tasks. Same frozen Claude baseline. Same judge.

Score: 80.0/100—89.4% of Claude, up from 25%. By category, the 122B model beat Claude on general (90 vs 85), landed within a few points on notes (92 vs 97) and email (87 vs 92), and trailed on files (64 vs 88, the most tool-chaining-heavy category). Reliability transformed: malformed tool calls went from 7 of 27 to zero. Tool-overlap recall doubled to 38.0% (still clearly behind Claude's implicit choices, but no longer broken). One production task—pilot-12, a code dump of 336,906 characters expanding to 281,190 tokens when encoded—still overflowed the 256K context window by roughly 25,000 tokens; Ollama rejected it precisely.

Cost is the other half. Claude's side came from real Langfuse API billing. The local side was metered per-task electricity using HomeLab Monitor (the author's open-source stack), attributed to measured GPU energy and converted at the author's actual dual-rate tariff. Neither is an estimate. The result: $0.000969 per task on local hardware versus $0.763 on Claude—roughly 787× cheaper. That gap was enough to move the dial: the author cancelled Claude Max and moved Jarvis to the local model, though keeping Claude Pro for other work. The honest caveat: model size and context budget changed together, with no isolated control, so the numbers measure a system upgrade rather than a clean model effect.