OWLET: On-Policy Weighted Learning via Evidence Transfer for Video Large Language Model Post-Training

Key points

• Existing post-training methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD).1
• RL supervises the policy with a single trajectory-level reward, which is too coarse-grained for video reasoning: the same response-level advantage is applied to every generated token, disregarding that the tokens of a video response are grounded in the visual stream to very different degrees.1
• Building on this observation, we propose OWLET, an evidence-privileged RL method that derives training signals from a richer teacher view.1
• To better exploit this privileged visual evidence, OWLET introduces (i) evidence-aware credit redistribution, which compares token likelihoods across the two views and redistributes positive response-level advantages according to their relative evidence support; and (ii) spatiotemporal state alignment, which aligns the two views' hidden states on verifier-accepted responses.1

Sources (1)

• [1]OWLET: On-Policy Weighted Learning via Evidence Transfer for Video Large Language Model Post-Training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 09:23 AM

  Post-training has become a key stage for eliciting reasoning capabilities in Video Large Language Models (Video-LLMs).

  Existing post-training methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD).

Extractive summary: sentences quoted from the sources.