Branching On-Policy Distillation
We propose BranchOPD (Branching On-Policy Distillation), which selects informative decision points using student uncertainty and teacher-student disagreement, resamples student-policy continuations from shared prefixes, and applies token-level teacher supervision to both main trajectories and branches.
ProofPaper ↗
Key points
- On-policy distillation (OPD) provides teacher supervision on student-generated trajectories, but a single trajectory realizes only one continuation at each decision, limiting supervision coverage under a finite sampling budget.
- BranchOPD operates at token-level decisions in mathematics and within-turn decisions in agent tasks.
- We also introduce protocol-boundary repair (PBR), which uses targeted supervision under protocol constraints to mitigate repetition and wasted generation.
- Mechanistic analyses and selection ablations show that branching at uncertain decisions expands subsequent behavior coverage and outperforms random selection and turn-initial branching.
Sources (1)
- [1]Branching On-Policy DistillationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:13 PM
We propose BranchOPD (Branching On-Policy Distillation), which selects informative decision points using student uncertainty and teacher-student disagreement, resamples student-policy continuations from shared prefixes, and applies token-level teacher supervision to both main trajectories and branches.
On-policy distillation (OPD) provides teacher supervision on student-generated trajectories, but a single trajectory realizes only one continuation at each decision, limiting supervision coverage under a finite sampling budget.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Continually Post-training Small Models via Synthetic Self-distillation
- Oct 9, 2026Rethinking Relative Policy Differences Transfer of On-Policy Distillation
- Oct 8, 2026One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts
- Oct 8, 2026SpatialOPSD: Self-Distilling Spatial Intelligence from Verified Coding Agent Traces
- Oct 8, 2026Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models
- Oct 8, 2026ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills