Decoupling Exploration from Optimization in RLVR
Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints.
Key points
- A key promise of RLVR is the discovery of new reasoning strategies.
- In principle, a model can sample novel ideas absent from its prior training data.
- Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis).
- Moreover, we observe improved pass@$k$ scaling, indicating that ExpDis produces models that generate more diverse correct solutions.
Sources (1)
- [1]Decoupling Exploration from Optimization in RLVRarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 05:59 PM
Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints.
A key promise of RLVR is the discovery of new reasoning strategies.
Extractive summary: sentences quoted from the sources.