Emergent Inverse-Depth Scaling From Nonlinearity In Attention

Key points

• Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood.1
• To explain the parameter count scaling, existing theory posits power-law scaling with model depth.1
• In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones.1
• Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.1

Sources (1)

• [1]Emergent Inverse-Depth Scaling From Nonlinearity In Attention
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 01:23 AM

  Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra.

  Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood.

Extractive summary: sentences quoted from the sources.