Rethinking Relative Policy Differences Transfer of On-Policy Distillation

Key points

• On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but typically relies on the teacher being substantially more capable than the target.1
• Recent work relaxes this requirement by transferring the relative policy difference between an expert and a reference policy rather than the expert's absolute distribution.1
• First, a stronger expert is not necessarily a better source of supervision: differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently, whereas those arising from model scale alone transfer weakly or not at all.1
• These results characterize when relative policy differences provide effective supervision for OPD.1

Sources (2)

• [1]Rethinking Relative Policy Differences Transfer of On-Policy Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 08:24 AM

  We develop a unified view in which the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization.

  On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but typically relies on the teacher being substantially more capable than the target.

• [2]Rethinking Relative Policy Differences Transfer of On-Policy Distillation
  Hugging Face Daily Papers · Oct 9, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.