Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Models

Key points

• Attention sinks are tokens or positions that consistently receive disproportionate attention mass, often beyond what their semantic content would suggest.1
• Despite extensive study in large language models, their behavior and functional relevance in text-to-image (T2I) Transformers remain largely unexplored.1
• Through systematic attention analysis and targeted interventions, we identify that attention sinks emerge in both bidirectional diffusion and autoregressive models but exhibit distinct patterns.1
• We further introduce register tokens to redistribute attention sinks and find that several configurations preserve or modestly improve generation performance.1

Sources (1)

• [1]Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 10:10 AM

  To fill this gap and better understand how text-to-image Transformers allocate and route attention, we explore this phenomenon of text-to-image transformers across bidirectional diffusion models and causal autoregressive models.

  Attention sinks are tokens or positions that consistently receive disproportionate attention mass, often beyond what their semantic content would suggest.

Extractive summary: sentences quoted from the sources.