Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models

Key points

• To address these limitations, we introduce LoopOPD, a cross-loop on-policy distillation framework that uses additional recurrent computation within a LoopLM as its own source of supervision.1
• We further propose Dynamic LoopOPD (D-LoopOPD), which continually refreshes the terminal loop teacher as the shared model parameters are updated, enabling recurrent self-improvement.1
• Experiments on Ouro-Thinking models show that LoopOPD improves mathematical reasoning, while D-LoopOPD yields further gains through dynamic teacher updates.1
• Despite being trained only on mathematical data, the resulting models also improve on general reasoning and code generation benchmarks, demonstrating that recurrent computation can serve as an effective source of supervision for LoopLMs. Our code and model checkpoints will be released upon acceptance.1

Sources (1)

• [1]Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 08:54 AM

  Looped Language Models (LoopLMs) offer a parameter efficient approach to scaling reasoning by reusing shared parameters across recurrent computation steps.

  To address these limitations, we introduce LoopOPD, a cross-loop on-policy distillation framework that uses additional recurrent computation within a LoopLM as its own source of supervision.

Extractive summary: sentences quoted from the sources.