AIToday
Large Language ModelsSimon Willison's WeblogPublished: Oct 6, 2026, 04:00 JST

Qwen3.8-27B-Q4_K_M.gguf scores 23.57% on word addition, 167/169 with reasoning

Send on LINE
Qwen3.8-27B-Q4_K_M.gguf scores 23.57% on word addition, 167/169 with reasoning

3 Key Points

  1. What happened

    Simon Willison replicated Colin Frasier's word-form addition test on a DGX Spark using Qwen3.8-27B-Q4_K_M.gguf. With reasoning disabled, 5,070 cases gave 23.57% numeric accuracy, falling from 97.04% on one- to three-digit operands to 6.44% on ten- to thirteen-digit ones.

  2. Why it matters

    The result suggests that disabling reasoning severely limits this model's ability to compute sums and express them in words, with accuracy crumbling as operand size grows, even though format compliance stayed at 96.17%.

  3. What to watch

    The improvement with reasoning enabled rested on just one attempt per pair, so a second run would likely differ; watch for a larger sample to confirm the gain.

WHO IT HITSTeams running local models on hardware like the DGX Spark may see that reasoning mode is essential for arithmetic tasks, while format compliance alone does not guarantee correct sums.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

Colin Frasier originally ran a similar experiment over two years ago using GPT-4o to see how well it could compute sums and return answers in words across increasingly large numbers. Simon Willison, confident that GPT-4o did not cheat with a calculator, was inspired to rerun the test on local hardware—a DGX Spark—for a fully controlled environment. He used Codex Remote (GPT-6 Astra) to run the same experiment with Qwen3.8-27B-Q4_K_M.gguf, first with reasoning disabled across 30 attempts per combination, then with reasoning enabled on a single attempt per pair. The reasoning-enabled run took much longer per pair, so only one sample per square was collected, producing a less visually detailed heatmap where each square is either 100% or 0%. The model got the right answer in 167 out of 169 attempts, and Willison notes that since these were one-shot, a second run would likely differ. The experiment highlights a sharp contrast between reasoning-disabled and reasoning-enabled performance on a task that requires both arithmetic and verbal formatting.

FAQ
What was the model's accuracy with reasoning disabled?
It achieved 23.57% numeric accuracy across 5,070 reasoning-disabled cases.
How did operand size affect performance?
Accuracy fell from 97.04% for one- to three-digit operands to 6.44% for ten- to thirteen-digit operands.
Did format compliance stay high?
Yes, it was 96.17% even when reasoning was disabled.
Simon Willison's WeblogRead Original Article
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleAnthropic staff gave $540 million in 2025, nearly 5× biggest Fortune 500 donors