Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

Key points

• Across code generation and mathematical reasoning, OPD with larger-scale teachers exhibits early loss plateaus, with an average final loss reduction of 25.1% after 200 updates, compared with 96.2% for self-RL teachers, obtained by further reinforcement learning (RL) training of the initial student.1
• To understand this difference, we analyze OPD as an idealized continuous-time dynamical system in the small-learning-rate limit.1
• Our training-log diagnostics associate these plateaus with an early decline in a gradient-based learning-signal proxy while substantial loss remains; these measurements do not establish why the underlying gradient weakens.1
• Across runs with and without loss plateaus, we observe small relative parameter changes (0.025-0.098%) and high similarity between the student's representations before and after OPD (linear CKA $>0.98$ across layers).1

Sources (1)

• [1]Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 04:52 AM

  On-policy distillation (OPD) enables effective capability transfer between language models, yet the mechanisms underlying its failures are not fully understood.

  Across code generation and mathematical reasoning, OPD with larger-scale teachers exhibits early loss plateaus, with an average final loss reduction of 25.1% after 200 updates, compared with 96.2% for self-RL teachers, obtained by further reinforcement learning (RL) training of the initial student.

Extractive summary: sentences quoted from the sources.