Why DiT Models Underperform as Representation Learners without Long Skip Connections

Key points

• Despite the advancement of diffusion backbones into Diffusion Transformers (DiT), it has been found in previous literature that DiT models tend to yield weaker features despite their stronger generation capability.1
• In this work, we show via controlled from-scratch training that long skip connections (LSCs) affect feature quality: a DiT with LSCs can produce better features.1
• Specifically, the input to a diffusion backbone contains both the latent image and the diffusion noise, where the diffusion noise, though appearing in the form of noises, actually reflects the information of the diffusion trajectory.1
• To make the final prediction of a noisy variable ($v$ or $ε$), the diffusion backbone needs both signals.1

Sources (1)

• [1]Why DiT Models Underperform as Representation Learners without Long Skip Connections
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 10:19 AM

  Diffusion feature, i.e., intermediate activations extracted from diffusion model backbones, has emerged as a promising approach for unifying generation and vision understanding (discrimination).

  Despite the advancement of diffusion backbones into Diffusion Transformers (DiT), it has been found in previous literature that DiT models tend to yield weaker features despite their stronger generation capability.

Extractive summary: sentences quoted from the sources.