PAGE: Parts-Aware Guidance for Co-Speech Gesture Portrait Video Generation

Key points

• We present PAGE (Parts-Aware GuidancE), a general, backbone-agnostic optimization framework for diffusion-based generation that enhances local spatial fidelity and global audio-visual consistency at both the training-objective and denoising-trajectory levels.1
• During training, we propose Parts-Specific Reweighting (PSR), which adaptively increases supervision in semantically important regions such as lips, eyes, and hands, improving visual fidelity in perceptually critical areas.1
• During inference, we introduce Consistency Classifier Guidance (CCG), an audio-conditioned latent classifier that identifies and corrects audio-visual inconsistencies in the generated video.1
• To enable large-scale evaluation, we construct the Chinese Announcer Speech (CAS) dataset, the first comprehensive Chinese benchmark for co-speech gesture portrait video generation.1

Sources (1)

• [1]PAGE: Parts-Aware Guidance for Co-Speech Gesture Portrait Video Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 09:48 AM

  Co-speech gesture video generation aims to synthesize realistic, audio-driven portrait videos from a single reference image and a speech signal.

  We present PAGE (Parts-Aware GuidancE), a general, backbone-agnostic optimization framework for diffusion-based generation that enhances local spatial fidelity and global audio-visual consistency at both the training-objective and denoising-trajectory levels.

Extractive summary: sentences quoted from the sources.