Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers

Key points

• Standard MHA implicitly learns a static, head-wise Euclidean similarity corresponding to the diagonal blocks of a unified cross-head metric matrix, discarding all off-diagonal interactions across heads.1
• First, it learns the full cross-head metric matrix, inducing a Mahalanobis-like similarity that explicitly captures inter-head interactions beyond the restricted block-diagonal structure of MHA.1
• We further introduce principled regularization strategies, most notably stack-wise parameter sharing, to ensure computational efficiency and stable optimization.1
• Extensive experiments demonstrate that DAMCHA-based Transformers consistently outperform competing baselines across a range of benchmark tasks.1

Sources (1)

• [1]Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 09:11 AM

  In this paper, we propose Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a simple and effective attention mechanism that generalizes Multi-Head Attention (MHA) through the lens of metric learning.

  Standard MHA implicitly learns a static, head-wise Euclidean similarity corresponding to the diagonal blocks of a unified cross-head metric matrix, discarding all off-diagonal interactions across heads.

Extractive summary: sentences quoted from the sources.