CERO: Where and When to Allocate Rollouts for RL Post-Training

Key points

• We instead study how to coordinate a finite rollout budget over the entire training horizon.1
• We formulate this problem using a concave surrogate utility of cumulative prompt exposure and introduce CERO, an online primal dual scheduler for prompt admission and budget pacing.1
• A compact Fenchel representation linearizes the dependence on cumulative exposure, while projected online gradient descent updates prompt-specific supporting slopes and a shared budget price using reward-variation feedback and budget deviations.1
• We establish pathwise guarantees for the surrogate allocation objective against fixed-rate and same-path time-varying benchmarks, with explicit terms for proxy discrepancy and rate variation.1

Sources (1)

• [1]CERO: Where and When to Allocate Rollouts for RL Post-Training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 08:40 AM

  Adaptive rollout methods for group-relative reinforcement learning typically allocate a fixed per-update budget across prompts.

  We instead study how to coordinate a finite rollout budget over the entire training horizon.

Extractive summary: sentences quoted from the sources.