Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization

Key points

• Our analysis reveals a selective reshaping of exploration: RL contracts behavior globally, yet diversifies successful trajectories, elicits success with fewer rollouts, and covers more of the latent task-valid solution space than supervised fine-tuning.1
• Inspired by this, we introduce DRIVE (Diversity-driven RL fIne-tuning for VLA gEneralization), which turns successful-behavior diversity into an explicit RL objective.1
• DRIVE groups rollouts under matched task conditions, compares their trajectories with temporal alignment, and derives a success-conditioned intrinsic reward from relative behavioral diversity.1
• Across LIBERO-Plus, ManiSkill3, and RoboTwin 2.0, DRIVE improves the average out-of-domain (OOD) performance over vanilla RL fine-tuning by 5.3 points on $π0$ and 2.0 points on $π{0.5}$.1

Sources (1)

• [1]Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 12:26 PM

  Reinforcement learning (RL) fine-tuning improves vision-language-action (VLA) policies through closed-loop experience, yet generalization beyond the fine-tuning distribution remains limited.

  Our analysis reveals a selective reshaping of exploration: RL contracts behavior globally, yet diversifies successful trajectories, elicits success with fewer rollouts, and covers more of the latent task-valid solution space than supervised fine-tuning.

Extractive summary: sentences quoted from the sources.