Multi-Stream Attention Residuals
Inspired by the success of multi-head attention along the sequence axis, we introduce Multi-Stream Attention Residuals (MSAR), an extension of AttnRes that generalizes this mechanism to multi-head depth attention, yielding substantial performance gains with minimal additional overhead.
ProofPaper ↗
Key points
- Attention Residuals (AttnRes) enables content-dependent retrieval over network depth, which can be viewed as single-stream softmax attention along the depth axis.
- MSAR draws stream diversity from sequence context through multi-scale causal convolutions, allowing different heads to retrieve historical representations through distinct contextual views while preserving the integrity of the hidden representation.
- Scaling-law experiments yield an iso-loss compute speedup of $1.58\times$ over the standard residual baseline, compared with $1.35\times$ for AttnRes and $1.30\times$ for mHC.
- These results make MSAR a promising residual architecture for large-scale pre-training.
Sources (2)
- [1]Multi-Stream Attention ResidualsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:13 PM
Inspired by the success of multi-head attention along the sequence axis, we introduce Multi-Stream Attention Residuals (MSAR), an extension of AttnRes that generalizes this mechanism to multi-head depth attention, yielding substantial performance gains with minimal additional overhead.
Attention Residuals (AttnRes) enables content-dependent retrieval over network depth, which can be viewed as single-stream softmax attention along the depth axis.
- [2]Multi-Stream Attention ResidualsHugging Face Daily Papers · Oct 9, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 8, 2026Cost-Aware Mixture-of-Experts Coordination for Model Markets
- Oct 8, 2026RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
- Oct 8, 2026MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement
- Oct 8, 2026One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts
- Oct 8, 2026One Model to Loop Them All: Elastic Latent Recurrence for Language Models
- Oct 7, 2026When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry
