When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

Key points

• On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models.1
• We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency.1
• We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios.1
• For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal.1

Sources (1)

• [1]When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:54 AM

  In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs?

  On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models.

Extractive summary: sentences quoted from the sources.