Diverse Motion Customization via Control-based Dynamic Optimization

Key points

• Despite recent advances in video generation, motion customization remains challenging due to content leakage, where appearance attributes from the reference video unintentionally propagate into the generated output.1
• We identify this issue as a consequence of the generative process collapsing toward the reference video, which arises from formulating the learning objective as a direct regression on the reference.1
• Our key idea is to steer generative dynamics toward desired motion while avoiding collapse toward the reference video, which we formalize using Stochastic Optimal Control (SOC).1
• Furthermore, to improve efficiency, we tailor the SOC formulation to motion customization by eliminating the need for an explicit reward and introducing a timestep-adaptive motion cost that focuses only on early generative stages, accelerating training by 2.5 times.1

Sources (1)

• [1]Diverse Motion Customization via Control-based Dynamic Optimization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 07:55 AM

  To address this, we propose Control-based Motion Customization (CMC), a principled training framework that is structurally robust to content leakage.

  Despite recent advances in video generation, motion customization remains challenging due to content leakage, where appearance attributes from the reference video unintentionally propagate into the generated output.

Extractive summary: sentences quoted from the sources.