Selective Transfer of RL Updates for Visual Reasoning

Key points

• We instead formulate capability transfer around the training-stage update, isolating the parameter changes induced by reinforcement learning (RL).1
• Yet transferring this update in full remains suboptimal: we find that its components differ substantially in cross-model transferability, with dominant directions transferring more effectively than the complete update.1
• Based on this finding, we introduce Selective-RL, which isolates the RL-stage update, retains its dominant matrix-wise directions with magnitude preservation, and transfers them to the language modules of a VLM.1
• Across three model families and five visual-reasoning benchmarks, Selective-RL improves full-update interpolation in 12 of 15 comparisons, including an 8.55 percentage-point MathVision gain on the Qwen recipient.1

Sources (1)

• [1]Selective Transfer of RL Updates for Visual Reasoning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 04:43 PM

  Model merging provides a training-free way to transfer reasoning capabilities from language models to vision-language models (VLMs), but endpoint-based transfer can conflate pre-existing model differences with changes acquired during reasoning post-training.

  We instead formulate capability transfer around the training-stage update, isolating the parameter changes induced by reinforcement learning (RL).

Extractive summary: sentences quoted from the sources.