Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking

Key points

• This paper shows that a natural account of these spectral phenomena emerges when the original setting is understood as a departure from a nearby, otherwise hidden, highly symmetric reference.1
• Modifications, including changes to the architecture, data distribution, or parameter metric, expose a nearby reference configuration whose Hessian exhibits rich invariances-ones not accounted for by weight symmetries.1
• The framework is developed in some generality, with a detailed analysis of three-layer ReLU networks and applications to convolutional, graph, and transformer models, as well as to the NTK.1
• The same mechanism is further shown to yield analogous spectral structures in layerwise Hessians and the Gauss-Newton matrix.1

Sources (1)

• [1]Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 12:08 PM

  Hessian spectra at trained models in deep learning exhibit a persistent pattern: eigenvalues organize into distinct clusters, including a large bulk near zero and a few isolated outliers.

  This paper shows that a natural account of these spectral phenomena emerges when the original setting is understood as a departure from a nearby, otherwise hidden, highly symmetric reference.

Extractive summary: sentences quoted from the sources.