Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding

Key points

• Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale.1
• Grounded-Dreamer combines the visual realism of generative video models with simulator physics engines.1
• Starting from a small set of real images depicting robot task initial states, our method automatically reconstructs task-relevant objects and camera geometry, aligns them in simulation, and collects physically valid trajectories.1
• Empirical results in real-world tasks demonstrate that Grounded-Dreamer can synthesize physically grounded data and substantially facilitate policy learning without requiring human teleoperation effort.1

Sources (1)

• [1]Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 07:53 AM

  In this paper, we introduce Grounded-Dreamer, a framework for synthesizing physically grounded robot manipulation data with video world models while avoiding human teleoperation demonstrations.

  Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale.

Extractive summary: sentences quoted from the sources.