Symmetry-Aware Feature Learning: A Polynomial Separation for Multi-Index Models

Key points

• We study growing-rank multi-index models with high-dimensional Gaussian covariates in $\mathbb{R}^d$ and $r=Θ(d^δ)$ teacher directions forming a cyclic symmetry orbit, where $0<δ<1/2$.1
• We compare three ways of exploiting this structure: architectural weight sharing, data augmentation over the full symmetry group, and learning without access to the symmetry.1
• In particular, we analyze a symmetry-tied convolutional network, an untied network, and the same untied network trained with full-group data augmentation, using spherical online SGD with correlation loss.1
• For a class of polynomial links with information exponent $p\ge3$, we prove matching sample complexity bounds up to logarithmic factors: the tied and augmented learners achieve weak directional recovery in $\widetildeΘ(d^{p-1})$ samples, whereas the symmetry-agnostic learner requires $\widetildeΘ(rd^{p-1})$.1

Sources (1)

• [1]Symmetry-Aware Feature Learning: A Polynomial Separation for Multi-Index Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 02:23 PM

  We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning.

  We study growing-rank multi-index models with high-dimensional Gaussian covariates in $\mathbb{R}^d$ and $r=Θ(d^δ)$ teacher directions forming a cyclic symmetry orbit, where $0<δ<1/2$.

Extractive summary: sentences quoted from the sources.