Leaner Transformers Can Easily Learn to Cluster

Key points

• Transformers have in-context learning capabilities, where some known learning algorithms can be executed in the forward pass through the model.1
• In this work, we build upon this result in the following ways: First, we present an equally expressive but smaller transformer that executes Lloyd's algorithm with embedding size $d{\textsf{emb}} = (d + \lceil \log2 k \rceil)$.1
• Next, we train these transformers to learn the clustering algorithms given a distribution of clustering tasks, and theoretically characterize and empirically validate the factors affecting the convergence and in-distribution generalization of learning algorithms based on stochastic gradients.1
• Finally, we probe the general clustering abilities of these learned algorithms (in the form of transformers), and try to understand situations where they succeed and fail.1

Sources (1)

• [1]Leaner Transformers Can Easily Learn to Cluster
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 09:46 AM

  Recent work shows that transformers can exactly perform Lloyd's algorithm for $k$-means clustering with $n$ points in $d$ dimensions with an embedding size $d_{\textsf{emb}} = d+k$ (thus, requiring attention projection matrices of size $(d+k)^2$).

  Transformers have in-context learning capabilities, where some known learning algorithms can be executed in the forward pass through the model.

Extractive summary: sentences quoted from the sources.