Two years ago Colin Frasier posted an experiment that tested GPT-4o’s ability to compute sums and return the answers in words as the magnitude of the numbers increased. He shared a chart showing the results of that run.
Inspired by that post, the author re-ran the experiment in a fully controlled local environment on an NVIDIA DGX Spark using the Qwen3.8-27B model.
Method
The chart image from Colin Frasier was pasted into a Codex Remote session (GPT-6 Astra), and the same experimental setup was executed with the Qwen3.8-27B-Q4_K_M.gguf model. Two main configurations were tested:
- Reasoning disabled: 30 attempts per number pair.
- Reasoning enabled: because this ran much slower, only one attempt per pair was executed (one-shot per square).
Results
- With reasoning disabled, performance was mixed and many calculations were incorrect, reinforcing the author’s belief that the model was not using an external calculator.
- With reasoning enabled in one-shot mode, the model produced correct answers in 167 out of 169 attempts. The author notes that since these were single-shot runs, a repeat could yield different outcomes.
Additional materials
A version of the report includes the reasoning traces for some of the larger additions. One example shows the model performing digit-by-digit addition from right to left, explicitly noting carries, for instance:
4,299,366,105,622 6,088,794,067,970
The trace aligns digits and adds each position (units, tens, hundreds...), with intermediate steps such as "6 + 9 = 15, write 5, carry 1."
Why it matters
The experiment demonstrates that a large language model running locally can produce correct, word-form numeric sums at high one-shot accuracy in many cases, and that enabling explicit reasoning affects both correctness and runtime. The result is relevant for applications that require concise textual numeric answers, but reproducibility and detailed methodological comparison warrant further study.
Tags
mathematics, ai, generative-ai, local-llms, llms, qwen, llm-reasoning, dgx-spark
