RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer
We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning.
ProofPaper ↗
Key points
- Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies.
- RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors.
- On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality.
- In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively.
Sources (1)
- [1]RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real TransferarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 12:39 AM
We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning.
Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies.
Extractive summary: sentences quoted from the sources.