I trained a 414k-parameter transformer to fly a boids flock, then tested whether the rules a probe can read are the ones it uses [P]

Key points

• (built in Python and PyTorch: a transformer written from scratch, one token per bird, two attention layers, about 414k weights, trained on a laptop CPU)1
• for comparison, a small network that only gets a bird's own move plus the flock's average move (no attention, no positions) gets R2 0.40. one seed, same held-out clips.1
• the three rules were easy to read out of its hidden states with a linear probe. cohesion (stay close to your neighbours) could be read before any training (probe R2 about 0.85).1
• what I am still not sure about: making a layer attend only to itself is something the model never went through during training. is that a fair test, or is some of the decay due to strange input fed to the model?1

Sources (1)

• [1]I trained a 414k-parameter transformer to fly a boids flock, then tested whether the rules a probe can read are the ones it uses [P]
  r/MachineLearning (top, daily) · Oct 11, 02:25 PM

  I wrote a small boid simulator (12 birds), recorded it flying, and trained a transformer to predict each bird's next move without it knowing about any boid rules.

  (built in Python and PyTorch: a transformer written from scratch, one token per bird, two attention layers, about 414k weights, trained on a laptop CPU)

Extractive summary: sentences quoted from the sources.