
What happened
Simon Willison replicated Colin Frasier's word-form addition test on a DGX Spark using Qwen3.8-27B-Q4_K_M.gguf. With reasoning disabled, 5,070 cases gave 23.57% numeric accuracy, falling from 97.04% on one- to three-digit operands to 6.44% on ten- to thirteen-digit ones.
Why it matters
The result suggests that disabling reasoning severely limits this model's ability to compute sums and express them in words, with accuracy crumbling as operand size grows, even though format compliance stayed at 96.17%.
What to watch
The improvement with reasoning enabled rested on just one attempt per pair, so a second run would likely differ; watch for a larger sample to confirm the gain.
WHO IT HITSTeams running local models on hardware like the DGX Spark may see that reasoning mode is essential for arithmetic tasks, while format compliance alone does not guarantee correct sums.
Summaries like this, every morning.
Colin Frasier originally ran a similar experiment over two years ago using GPT-4o to see how well it could compute sums and return answers in words across increasingly large numbers. Simon Willison, confident that GPT-4o did not cheat with a calculator, was inspired to rerun the test on local hardware—a DGX Spark—for a fully controlled environment. He used Codex Remote (GPT-6 Astra) to run the same experiment with Qwen3.8-27B-Q4_K_M.gguf, first with reasoning disabled across 30 attempts per combination, then with reasoning enabled on a single attempt per pair. The reasoning-enabled run took much longer per pair, so only one sample per square was collected, producing a less visually detailed heatmap where each square is either 100% or 0%. The model got the right answer in 167 out of 169 attempts, and Willison notes that since these were one-shot, a second run would likely differ. The experiment highlights a sharp contrast between reasoning-disabled and reasoning-enabled performance on a task that requires both arithmetic and verbal formatting.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
Researchers at the Technion, University of Haifa, and Google Research released AltRA-Test, which checks whethe…

The Nov 19 arXiv paper found ICLR critique overlap rose from 12.0% in 2026 to 21.3% in 2027, with LLM-flagged…

Feed Inc. began offering "Marugoto LLMO" on November 17, 2026, covering AI research, question design, page rew…

Co-Labo AI published the first of three explainer installments on ontology on November 20, 2026, aimed at manu…

The guide maps input types to fields — images and video to computer vision (CV), text to natural language proc…

Across 52 laboratory tasks in physical sciences, life sciences and engineering, Claude Fable 5.2 led 13 agents…
