Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers

Key points

• Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear.1
• However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context processing.1
• Our design grants Orthros attention a lower asymptotic complexity compared to vanilla Transformers in prevalent long-context scenarios characterized by heavy prefilling and light decoding.1
• Finally, Orthros achieves a $21.9\times$ prefilling speedup over Transformers with end-to-end inference gains, while preserving long-context capabilities, as demonstrated by near-perfect 128K needle-in-a-haystack results and Transformer-level performance on real-world in-context recall tasks.1

Sources (1)

• [1]Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 03:30 PM

  To address this, we introduce Orthros attention, which shifts the efficiency paradigm from structural allocation to phase-aware decoupling.

  Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear.

Extractive summary: sentences quoted from the sources.