Dream to Remember: Continual Steering of Generalist Robot Policies with World Models

Key points

• Reinforcement learning can improve and extend them, either by fine-tuning the policy weights or by training a small module to steer the input noise of a frozen generative policy.1
• We propose Dream to Remember (D2R), which continually steers a frozen generalist policy toward new tasks, while retaining its earlier skills.1
• As the steering module learns a new task, a frozen pretrained world model rehearses prior tasks through imagined rollouts, anchoring the module to its earlier behavior without a replay buffer or new data.1
• With a frozen off-the-shelf generalist policy and world model, $π0$ and A2World, D2R learns unseen LIBERO-90 tasks in the environment while keeping $π0$'s performance on the LIBERO suites it was fine-tuned on, even though the world model is not accurate enough for policy optimization.1

Sources (1)

• [1]Dream to Remember: Continual Steering of Generalist Robot Policies with World Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 04:28 PM

  Generalist robot policies are trained with imitation learning and are bounded by the quality and coverage of their demonstrations.

  Reinforcement learning can improve and extend them, either by fine-tuning the policy weights or by training a small module to steer the input noise of a frozen generative policy.

Extractive summary: sentences quoted from the sources.