Does On-Policy Distillation for Safety Pose Backdoor Risks?
On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models.
ProofPaper ↗
Key points
- Recent studies further explore OPD as a tool for improving large language model safety with promising results.
- In this paper, we uncover an overlooked threat to OPD for safety: a safety-aligned but backdoored teacher can propagate its hidden malicious behavior to an initially clean student.
- Second, the commonly used top-k KL can accelerate backdoor transfer, causing trigger-conditioned harmful behavior to emerge earlier than sampled-token KL in most settings.
- Experiments show that Lazy Defense delays backdoor transfer in low poisoning rate settings.
Sources (1)
- [1]Does On-Policy Distillation for Safety Pose Backdoor Risks?arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 02:50 AM
On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models.
Recent studies further explore OPD as a tool for improving large language model safety with promising results.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 6, 2026Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight
- Oct 6, 2026RISED: Rubrics for Agentic Multi-Environment Selection and Self-Distillation
- Sep 29, 2026[AINews] AMD buys World Labs for $8.2B, as Atlas solves sparse reconstruction problem for robotics, design and more
- Sep 29, 2026Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation
- Sep 28, 2026Notes on NVIDIA Nemotron
- Aug 10, 2026huggingface/transformers v5.15.0: Release: v5.15.0