Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation

Key points

• Reinforcement learning (RL) and sampling-based model predictive control (MPC) are commonly employed to yield dynamically feasible motions, but both are sample-inefficient and sensitive to hyperparameters.1
• We hypothesize that dynamically feasible trajectories concentrate near a low-dimensional manifold shared across demonstrations, so that retargeting can be reduced to sampling from that manifold, conditioned on human motion, rather than solving a fresh optimization problem for every demonstration.1
• We propose Generative Neural Retargeting (GNR), which uses a flow matching model to sample feasible trajectories.1
• GNR can be used for scalable and efficient retargeting of large-scale, long-horizon, and millimeter precision human demonstrations: by applying GNR within a real-to-sim data engine, we produce a dexterous manipulation dataset with dense contact-force labels, spanning $223$k demonstrations and $3.3$k object geometries.1

Sources (1)

• [1]Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:57 PM

  Human demonstrations are a scalable data source for learning dexterous manipulation, but the embodiment gap prevents human motion from being executed directly on robots.

  Reinforcement learning (RL) and sampling-based model predictive control (MPC) are commonly employed to yield dynamically feasible motions, but both are sample-inefficient and sensitive to hyperparameters.

Extractive summary: sentences quoted from the sources.