RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models

Key points

• This overhead is particularly pronounced during long-context prefill, where the full input sequence is repeatedly processed across recurrent depth.1
• Weight-and-activation quantization offers a natural way to accelerate this computation through lower-precision arithmetic, yet repeatedly applying quantized computation introduces numerical perturbations that accumulate along the recurrent trajectory.1
• We systematically analyze how such errors propagate across recurrent depth and show that quantization errors introduced earlier are attenuated by more subsequent recurrent steps and therefore have a smaller effect on the final representation.1
• Based on this insight, we introduce RAMP (Recurrent Adaptive Mixed Precision), which performs early recurrent refinement using low-precision weight-and-activation computation and restores high precision for terminal refinement.1

Sources (1)

• [1]RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 05:57 AM

  Recurrent-depth language models improve inference-time reasoning by repeatedly applying a shared network to refine internal representations, but the additional recurrent computation also increases inference cost.

  This overhead is particularly pronounced during long-context prefill, where the full input sequence is repeatedly processed across recurrent depth.

Extractive summary: sentences quoted from the sources.