MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

Key points

• Data-centric scaling of RLVR has focused on collecting or generating verifiable training problems.1
• To guide the synthesis of effective RLVR data, we experimentally characterize dataset effectiveness as a function of its problems, the target policy, and the training rollout budget.1
• Guided by these empirical principles, we introduce MagicBeanStalk, a top-down decomposition and bottom-up recursive data-synthesis algorithm that grows seed problems into families of substantively new, policy- and budget-calibrated training problems while preserving the seeds' reasoning blueprints.1
• Across 8 math benchmarks, continual RLVR on MagicBeanStalk problems improves gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4Flash from 72.4 to 77.9 (+5.4) on a 16K-token budget.1

Sources (1)

• [1]MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 07:25 PM

  Under this characterization, we show that a large open-source math corpus pooled from widely used RLVR datasets falls almost entirely into a zero-advantage trap: for a fixed target policy and rollout budget, roughly 88% of its problems are solved in every rollout or failed in every rollout.

  Data-centric scaling of RLVR has focused on collecting or generating verifiable training problems.

Extractive summary: sentences quoted from the sources.