Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization

Key points

• We redesign 4-bit optimizer-state quantization for AdamW from the perspective of rounding space: the coordinate in which a quantizer chooses between adjacent reconstruction levels.1
• For the second moment, a local analysis of the quantization cell adjacent to zero shows that small mean state error need not imply small mean preconditioner error at the next step.1
• A one-dimensional quadratic construction further shows qualitatively different optimization dynamics under state-space and preconditioner-space rounding.1
• These results motivate Zero-Inclusive Preconditioner-space Stochastic Rounding (ZIP-SR), which retains zero in the second-moment codebook and computes stochastic-rounding probabilities in preconditioner space.1

Sources (1)

• [1]Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:58 PM

  Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates.

  We redesign 4-bit optimizer-state quantization for AdamW from the perspective of rounding space: the coordinate in which a quantizer chooses between adjacent reconstruction levels.

Extractive summary: sentences quoted from the sources.