Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers

Key points

• Even though naturally discrete-time, this dynamical system is most often studied in continuous-time.1
• Scope of this paper is to provide a thorough investigation of the discrete-time version of the self-attention dynamics: a complete classification of its equilibria is given, and their stability properties are studied, in both the single-head and the multihead case.1
• We show that in both cases the main attractor of the dynamics is a consensus equilibrium aligned with the principal eigenvector of the value matrix (in the single-head case, a corresponding matrix of weights in the multihead case), even though other locally asymptotically stable bipartite equilibria can be show to exist at least in the single-head situation.1
• The behavior is confirmed by looking at the learned weight matrices of a real transformer.1

Sources (1)

• [1]Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:39 PM

  In machine learning, a self-attention dynamics describes the attention mechanism of transformers, and is obtained treating the layer index of a transformer as time index.

  Even though naturally discrete-time, this dynamical system is most often studied in continuous-time.

Extractive summary: sentences quoted from the sources.