Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers

Key points

• Autoregressive Large Language Models (LLMs) frequently struggle with deterministic multi-step algorithmic tasks such as multi-digit multiplication and long division.1
• Introducing a deterministic Digit-by-Digit Long Division scratchpad within a 4-stage Hierarchical Developmental Curriculum dramatically elevates single-digit division from 4.0% to 86.7% accuracy on a 4,000-problem held-out benchmark.1
• In contrast, multi-digit multiplication remained challenging: detailed error analysis revealed that while the model correctly computed single-digit sub-products and place-value zeros, our FOIL scratchpad failed because it forced a simultaneous summation of up to nine multi-digit terms in a single step without pairwise intermediate accumulation.1
• Finally, we identify two key boundaries: performance collapses to 0.00% on unseen 4-digit operands, and unbuffered training induces catastrophic forgetting, collapsing division accuracy from 86.7% down to 0.00%.1

Sources (1)

• [1]Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 06:58 PM

  In this paper, we investigate the mechanics of multi-step arithmetic in compact "Tiny" Transformers ( 10.6M non-embedding parameters, 49.3M total) trained on synthetic data across four basic operations (+, -, *, /) unrolled as step-by-step scratchpads.

  Autoregressive Large Language Models (LLMs) frequently struggle with deterministic multi-step algorithmic tasks such as multi-digit multiplication and long division.

Extractive summary: sentences quoted from the sources.