Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Study

Key points

• In this work, we conduct the first rigorous analysis of feature information across diverse architectural scales, empirically uncover the relationship between ViT representation and generalization behavior, and leverage these insights to guide efficient ViT design.1
• Our contributions are fivefold: Across diverse architectural scales, 1) We identify feature collapse at initialization, which leads to redundancy, and propose a reduction scheme to mitigate this issue.1
• 3) We show that feature in the token space provides a more faithful representation than those in embedding space.1
• 4) We discover an unexpected finding: features produced by linear submodules within ViT layers are critical for the prediction of generalization performance.1

Sources (1)

• [1]Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Study
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 04:06 AM

  Representation structure is crucial for understanding Vision Transformer (ViT) architectures and their generalization behavior.

  In this work, we conduct the first rigorous analysis of feature information across diverse architectural scales, empirically uncover the relationship between ViT representation and generalization behavior, and leverage these insights to guide efficient ViT design.

Extractive summary: sentences quoted from the sources.