An all-transistor physical-learning transformer can be trained to perform in-context learning

Key points

• Most systems trained by equilibrium propagation or coupled learning are in the same class as multi-layer perceptrons: while they may be universal function approximators, they struggle to learn associative tasks which are easy for transformers.1
• Equilibrium propagation relies on a bidirectional flow of information through the physical system between input and output.1
• Here we explicitly restore this bidirectional flow by introducing extra circuit elements for this purpose into a double or dyadic circuit.1
• We then use these ideas to construct an attention mechanism exploiting the exponential current law of sub-threshold MOSFET devices.1

Sources (1)

• [1]An all-transistor physical-learning transformer can be trained to perform in-context learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 06:01 PM

  We show that in simulations this circuit can be trained to perform a simple in-context learning task, despite using realistic reverse components which transmit only an approximate gradient.

  Most systems trained by equilibrium propagation or coupled learning are in the same class as multi-layer perceptrons: while they may be universal function approximators, they struggle to learn associative tasks which are easy for transformers.

Extractive summary: sentences quoted from the sources.