Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks

Key points

• While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear.1
• We address this question using probability signatures, which express leading gradient interactions through conditional statistics of the training distribution.1
• For modular addition, these signatures are cyclic shift operators and are diagonalized by the discrete Fourier transform, yielding approximately decoupled Fourier-mode dynamics.1
• We show that noise increases conditional label collisions, strengthening early shared-coordinate reinforcement.1

Sources (1)

• [1]Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 12:27 PM

  Neural networks trained on modular addition tasks often develop Fourier-structured representations that support exact generalization.

  While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear.

Extractive summary: sentences quoted from the sources.