Adapting Vision-Language-Action Models to Unknown Visual Disruptions During Execution

Key points

• Visual disruptions can arise while a robot is executing a task, leaving a vision-language-action (VLA) policy to respond without knowing the disruption type or timing.1
• Because consecutive chunks overlap in time, the leftover provides a temporally aligned target for the current prediction over the same future control interval.1
• At the onset of a visual shift, the leftover can retain a plan formed before the corruption, so updating the policy toward it anchors the adaptation across the shift (Transition Anchoring).1
• On LIBERO-10, SALT increases average success across five persistent visual corruptions from 43.9% to 53.2% with SmolVLA and from 58.7% to 66.0% with GR00T N1.7, while largely preserving nominal performance.1

Sources (1)

• [1]Adapting Vision-Language-Action Models to Unknown Visual Disruptions During Execution
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 08:21 AM

  We introduce Self-supervised Adaptation from Leftover Trajectories (SALT), which uses the leftover trajectory, the unexecuted part of the previous action chunk, as self-supervision for test-time adaptation.

  Visual disruptions can arise while a robot is executing a task, leaving a vision-language-action (VLA) policy to respond without knowing the disruption type or timing.

Extractive summary: sentences quoted from the sources.