Fluctuations of Nonlinear Observables in Mean Field Neural Network Training
In this work, we show how these mean field fluctuations propagate to finite dimensional nonlinear observables for shallow neural networks trained by stochastic gradient descent.
ProofPaper ↗
Key points
- Mean field limits describe the training dynamics of wide neural networks through the evolution of the empirical distribution of their parameters.
- Working in the weighted Sobolev space in which the limiting fluctuation process is constructed, we apply a functional Delta method under ordinary Fr{é}chet differentiability, without requiring Lions derivatives with respect to the measure variable.
- We obtain a central limit theorem for the observables and, under a suitable representation of their differentials, an explicit covariance formula inherited from the underlying mean field fluctuation theory.
- Under a constant rank assumption, we prove that a quantity of interest factors locally through the observation functional if and only if, throughout a neighborhood, the kernel of the differential of the observation is contained in that of the quantity of interest.
Sources (1)
- [1]Fluctuations of Nonlinear Observables in Mean Field Neural Network TrainingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 09:51 AM
In this work, we show how these mean field fluctuations propagate to finite dimensional nonlinear observables for shallow neural networks trained by stochastic gradient descent.
Mean field limits describe the training dynamics of wide neural networks through the evolution of the empirical distribution of their parameters.
Extractive summary: sentences quoted from the sources.