Stochastic Teacher Intervention for Agentic On-Policy Distillation
To address this issue, we introduce STI-OPD, a stochastic teacher intervention framework for multi-turn agentic OPD.
ProofPaper ↗
Key points
- On-policy distillation (OPD) efficiently transfers capabilities from a stronger teacher to a student language model through dense token-level supervision on student-generated rollouts and has shown promise on complex tasks such as mathematical reasoning.
- During multi-turn interaction, STI-OPD uses teacher intervention guided by teacher-student policy discrepancy to replace the student's proposed action with a teacher-generated one to maximize the acquisition of reliable supervision.
- By sampling whether to intervene from this probability, STI-OPD adaptively balances teacher control with student exploration.
- To learn from the resulting mixed-policy trajectories, we introduce an Importance-Weighted Reverse KL objective that corrects the token sampling mismatch between teacher-generated responses and the student policy to preserve the original OPD objective.
Sources (1)
- [1]Stochastic Teacher Intervention for Agentic On-Policy DistillationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 08:29 PM
To address this issue, we introduce STI-OPD, a stochastic teacher intervention framework for multi-turn agentic OPD.
On-policy distillation (OPD) efficiently transfers capabilities from a stronger teacher to a student language model through dense token-level supervision on student-generated rollouts and has shown promise on complex tasks such as mathematical reasoning.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 7, 2026MIMESIS: Learning User Simulators as Training Environments for Interactive Agents
- Oct 7, 2026Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts
- Oct 7, 2026On-Policy Distillation Teaches New Skills but Not New Knowledge
- Oct 7, 2026UniSkill: Learning Actor-Aligned Skill Proposals for an Evolving Policy
- Oct 6, 2026Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight
- Oct 6, 2026RISED: Rubrics for Agentic Multi-Environment Selection and Self-Distillation