Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata

Key points

• Induction heads provide a mechanistic account of in-context learning in sequential data, but existing theory largely assumes that the context relevant to a prediction forms a contiguous block.1
• In multidimensional data, serialization breaks this assumption by scattering spatial neighbors across distant positions in the token sequence.1
• We study how transformers overcome this routing problem in multidimensional stochastic and deterministic cellular automata, where each trajectory is generated by an unknown local rule and presented as a flattened sequence without an explicit coordinate-based spatial inductive bias.1
• Attention patterns and layerwise probes align with the predicted gather-and-match computation, providing mechanistic evidence for spatial induction in trained transformers.1

Sources (1)

• [1]Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 09:16 PM

  We introduce spatial induction heads, two-layer gather-and-match circuits in which the first layer reconstructs the relevant spatial neighborhood and the second matches the resulting configuration against earlier occurrences.

  Induction heads provide a mechanistic account of in-context learning in sequential data, but existing theory largely assumes that the context relevant to a prediction forms a contiguous block.

Extractive summary: sentences quoted from the sources.