ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models

Key points

• Vision-Language-Action (VLA) models have become a central paradigm for robot policy learning, which predict actions in three forms: raw action chunks, discrete action tokens, or continuous action latents.1
• However, existing action representations primarily model action trajectories, with limited consideration of the visual dynamics induced by these actions.1
• Specifically, ViDAL learns action latent space by training an Action Variational Autoencoder (Action VAE) to reconstruct action chunks while aligning its latent with future scene dynamics.1
• Empirically, ViDAL outperforms competitive baselines on LIBERO with 98.1% average success, improves a multi-task $π{0.5}$ policy on RoboTwin 2.0 from 54.3% to 65.5% (Clean) and from 33.2% to 43.1% (Random) success rates over 50 dual-arm tasks, and yields 20.0% and 23.4% absolute success-rate gains on real-world single-arm Franka and dual-arm ARX robot platforms.1

Sources (1)

• [1]ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 11:02 AM

  We introduce ViDAL, a Visual Dynamics-grounded Action Latent Space that anchors continuous action latents in the future visual dynamics of the scene.

  Vision-Language-Action (VLA) models have become a central paradigm for robot policy learning, which predict actions in three forms: raw action chunks, discrete action tokens, or continuous action latents.

Extractive summary: sentences quoted from the sources.