CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization
In this paper we seek to formalize the process of going from synthetic video generation to real-world robot trajectories for zero-shot manipulation tasks.
ProofPaper ↗
Key points
- Prior research on learning from humans often requires real human demonstrations.
- But advances in video generation models offer a new paradigm: we can now prompt these generators to make synthetic videos of the task.
- On the plus side, these synthetic videos take place in the robot's current environment, and do not require actual human teaching.
- Specifically, we show that the video generation must probabilistically seed the search within a semantic basin of attraction where the trajectory can be locally modified without losing its task alignment.
Sources (1)
- [1]CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained OptimizationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 06:00 PM
In this paper we seek to formalize the process of going from synthetic video generation to real-world robot trajectories for zero-shot manipulation tasks.
Prior research on learning from humans often requires real human demonstrations.
Extractive summary: sentences quoted from the sources.