How Do Transformers Learn to Represent Symmetries?

Key points

• In this paper, we study the ability of a vanilla Transformer to learn various symmetries through finite data augmentation for point cloud datasets.1
• We identify an ordering of increasing learnability across the following symmetry groups: (i) non-angle-preserving symmetries, (ii) angle-preserving symmetries, and (iii) base angle-preserving subgroups, such as translation, rotation, and scale.1
• For the base angle-preserving groups, we further investigate the Transformer's extrapolation behavior and conduct a structural analysis of the trained models, allowing us to identify interpretable mechanisms that induce invariance.1
• Finally, we extend our analysis to equivariant functions and show that the detected mechanisms for approximate invariance can also provide a key building block for learned equivariance.1

Sources (1)

• [1]How Do Transformers Learn to Represent Symmetries?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 04:01 PM

  Training Transformer-based architectures with finite data augmentation has become an increasingly popular approach in geometric machine learning.

  In this paper, we study the ability of a vanilla Transformer to learn various symmetries through finite data augmentation for point cloud datasets.

Extractive summary: sentences quoted from the sources.