On-Policy Distillation with Negative-Policy Rollouts

Key points

• On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts.1
• Rather than modifying the distillation reward formulation, NP-OPD introduces the negative policy at the rollout stage, continuously supplying tokens preferred by the negative policy over the teacher so that they remain exposed to teacher supervision throughout training.1
• Through extensive experiments, we show that NP-OPD improves OPD across model scales, generation modes, reasoning domains, and different OPD variants.1
• These results support our design of introducing negative signals through negative-policy rollouts and provide new insight into the role of the rollout policy in OPD.1

Sources (1)

• [1]On-Policy Distillation with Negative-Policy Rollouts
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 07:25 AM

  In this work, we introduce Negative-Policy OPD (NP-OPD), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy that serves as a negative reference for the student.

  On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts.

Extractive summary: sentences quoted from the sources.