Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels

Key points

• Linearizing softmax attention offers a route to efficient Transformer inference, but preserving pretrained attention remains challenging.1
• We identify an important source of approximation error overlooked by standard linearization methods: a small number of outlier dimensions contribute disproportionately to query-key dot products and account for a large fraction of the approximation error.1
• Starting from a second-order Taylor approximation, we introduce a Hadamard-product formulation that makes individual query-key contributions explicit.1
• These results demonstrate the value of outlier-aware dimension grouping for both attention approximation and pretrained-model conversion.1

Sources (1)

• [1]Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:40 PM

  Motivated by this heterogeneity, we propose Outlier-Aware Attention Linearization (OAL), a dimension-aware method for constructing quadratic attention approximations.

  Linearizing softmax attention offers a route to efficient Transformer inference, but preserving pretrained attention remains challenging.

Extractive summary: sentences quoted from the sources.