Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models

Key points

• We ask how internal supervision can further support temporal learning.1
• Direct hidden-state alignment does not explicitly use the student's temporal responses to choose a supervision direction.1
• We propose Temporal-Response Guidance (TRG), which measures representation responses to local temporal perturbations, adds the teacher correction's projection onto their span back to the original correction, and restores its original norm.1
• With an orthogonal tangent projector and a detached target, TRG changes the supervision direction while preserving the cosine-loss value and aligned-representation gradient norm at target construction.1

Sources (1)

• [1]Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 06:01 AM

  On-policy distillation (OPD) transfers capabilities through teacher predictions on student-generated sequences, as studied by GKD and MiniLLM for language models and Video-OPD for video tasks.

  We ask how internal supervision can further support temporal learning.

Extractive summary: sentences quoted from the sources.