Force without transmission: a depth-induced rank collapse that no loss on the representation reopens
Training can drive a transformer into a rank collapse: all token representations point in one direction, and learning stops.
ProofPaper ↗
Key points
- In a related collapse of attention, a loss term with a bounded corrective force repairs the network during the run.
- We ask whether such a term repairs rank collapse.
- We collapse small transformers by weakening their skip connection and treat copies of the collapsed network.
- Whether a collapsed network can be repaired depends on whether the gradient still reaches the weights that must change, not on how strongly a loss term pushes.
Sources (1)
- [1]Force without transmission: a depth-induced rank collapse that no loss on the representation reopensarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 12:32 PM
Training can drive a transformer into a rank collapse: all token representations point in one direction, and learning stops.
In a related collapse of attention, a loss term with a bounded corrective force repairs the network during the run.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 7, 2026Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning
- Oct 6, 2026EmbeddingGemma 2: an open, lightweight multimodal embedding model
- Oct 5, 2026LiquidAI/d1-omni-600M
- Oct 5, 2026MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
- Sep 30, 2026Cloudflare/clef-flash
- Sep 29, 2026microsoft/AesCode-32B