Tracing the Thoughts of a Coding Agent Playing ARC-AGI-3: Lessons for Continual Learning

Key points

• The agent runs on a frozen foundation model inside a fixed harness and acts by writing and running Python and shell scripts.1
• It retains no state across turns other than its written artifacts, so every thought it forms, carries, corrects or abandons leaves a trace, where a thought is any belief, rule or plan committed to a file.1
• We let the agent play ARC-AGI-3, a set of interactive reasoning games that provide no instructions.1
• Our contribution is a measurement protocol that traces each thought through these files, from the task where it forms to the task where it is corrected or abandoned, applied to seven evaluation runs with three backbones from two model families.1

Sources (1)

• [1]Tracing the Thoughts of a Coding Agent Playing ARC-AGI-3: Lessons for Continual Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 08:04 AM

  We study how a coding agent learns across a sequence of abstract reasoning tasks.

  The agent runs on a frozen foundation model inside a fixed harness and acts by writing and running Python and shell scripts.

Extractive summary: sentences quoted from the sources.