Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

Key points

• In this paper, we argue that looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows.1
• We observe that increasing loop iterations can reduce reasoning accuracy: noisy state updates overwrite correct intermediate deductions and even undo completed solutions.1
• InfiLoop combines content-based weighting with learned temporal decay to maintain a running summary of recurrent states.1
• Across extensive reasoning tasks, a 7M-parameter InfiLoop model outperforms existing recursive architectures, reaching 97.9% exact accuracy on Sudoku-Extreme, and 13.6% pass@2 on ARC-AGI-2.1

Sources (2)

• [1]Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
  Hugging Face Daily Papers · Oct 8, 12:00 AM

  In this paper, we introduce InfiLoop, a loop-native residual connection that learns which past computations to retain and how much to accept from each new update.

  In this paper, we argue that looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows.

• [2]Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 09:26 AM · same content

Extractive summary: sentences quoted from the sources.