S$^3$R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding

Key points

• For group-relative optimization, we show that even exact response-wise reweighting does not generally recover the expected dense-policy update because the advantage of each response depends on the rewards of other group members.1
• We propose Self-Sparse Speculative Rollouts (S$^3$R).1
• A KV-aware scheduler therefore limits KV-cache repeated offloading and restoration while maintaining rollout parallelism. % OR thrashing In a matched ablation with an aggressive sparse drafting budget, S$^3$R maintains stable training while direct sparse sampling with training-side correction becomes unstable.1
• On long-context mathematical reasoning, S$^3$R achieves $1.39\times 1.35\times$ the end-to-end training throughput of autoregressive rollouts on Qwen3 model, with comparable task performance.1

Sources (1)

• [1]S$^3$R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 08:22 AM

  Fast rollout generation for long-context reinforcement learning with verifiable rewards (RLVR) requires reducing KV-cache traffic without changing the response groups used for policy optimization.

  For group-relative optimization, we show that even exact response-wise reweighting does not generally recover the expected dense-policy update because the advantage of each response depends on the rewards of other group members.

Extractive summary: sentences quoted from the sources.