Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping
Recent works on decentralized non-convex optimization have studied both clipping and normalization under heavy-tailed noise, with clipping yielding suboptimal rates and normalization needing local momentum or mini-batches to converge.
ProofPaper ↗
Key points
- Heavy-tailed noise has been widely observed in modern machine learning, motivating the use of methods like gradient clipping and normalization.
- This raises the question: can a baseline decentralized method using a nonlinearity achieve optimal convergence rates under heavy-tailed noise?
- For smooth non-convex costs under bounded $p$-th moment noise, $p \in (1,2]$, we show that clipped $\mathtt{DSGD}$ achieves order-optimal rates both with high probability and in expectation.
- Our results highlight an important distinction between clipping and normalization in decentralized settings: while normalized $\mathtt{DSGD}$ can fail to converge, clipping retains magnitude information, enabling $\mathtt{DSGD}$ to be convergent and order-optimal.
Sources (1)
- [1]Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient ClippingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 05:57 PM
Recent works on decentralized non-convex optimization have studied both clipping and normalization under heavy-tailed noise, with clipping yielding suboptimal rates and normalization needing local momentum or mini-batches to converge.
Heavy-tailed noise has been widely observed in modern machine learning, motivating the use of methods like gradient clipping and normalization.
Extractive summary: sentences quoted from the sources.