Connected Self Forcing: Beyond Local Learning in Video Autoregression

Key points

• We introduce Connected Self Forcing, a training framework that reconnects gradient paths across autoregressive chunks, allowing feedback from later predictions to guide how earlier context is generated.1
• These connections go beyond historical KV-writing: gradients pass through generated latents into the computations that produced them, linking the generation of earlier context to its use in later predictions.1
• To make this connected training memory-efficient, we develop shortcut gradient replay, which recovers cross-chunk gradients without retaining the full rollout computation graph.1
• Experiments on autoregressive video generation show improvements in long-horizon visual quality and temporal consistency, without changing the inference procedure.1

Sources (1)

• [1]Connected Self Forcing: Beyond Local Learning in Video Autoregression
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 03:36 PM

  To stream long videos while maintaining visual quality and temporal consistency, Self Forcing mitigates exposure bias through self-rollout training on self-generated histories with key-value (KV) caching.

  We introduce Connected Self Forcing, a training framework that reconnects gradient paths across autoregressive chunks, allowing feedback from later predictions to guide how earlier context is generated.

Extractive summary: sentences quoted from the sources.