RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning

Key points

• In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL scalability.1
• Asynchronous or partial-rollout methods improve throughput by relaxing synchronization, but inevitably introduce stale off-policy samples (trajectories) that may hurt final accuracy.1
• In this work, rather than passively reweighting samples during training, we propose RollVerify, a lightweight RL framework built on partial rollout that actively verifies and repairs samples before they enter training.1
• Experiments on mathematical and tool-assisted mathematical reasoning show that RollVerify achieves accuracy comparable to on-policy training while reducing training cost.1

Sources (1)

• [1]RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 12:05 PM

  Reinforcement learning is crucial for improving large language models' reasoning and generalization.

  In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL scalability.

Extractive summary: sentences quoted from the sources.