Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics

Key points

• Large-scale egocentric human videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning.1
• To address these limitations, we introduce HF-WM, an action-conditioned world model that combines Human interaction learning with Failure-aware post-training.1
• To bridge the embodiment gap, we develop a dual-branch framework that couples video generation with auxiliary pointflow prediction, using stabilized pointflow as a unified motion representation across human and robot interactions while suppressing egocentric camera motion.1
• To improve failure modeling, we further perform Group Relative Policy Optimization (GRPO) post-training on perturbed actions derived from successful demonstrations, guided by action alignment and physical plausibility rewards, enabling the model to learn plausible failure dynamics without paired ground-truth videos.1

Sources (1)

• [1]Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:44 PM

  Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions.

  Large-scale egocentric human videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning.

Extractive summary: sentences quoted from the sources.