Here the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation
Recent joint video-audio generation models have achieved strong semantic correspondence and temporal synchronization.
ProofPaper ↗
Key points
- We refer to this property as Dynamic Spatial Correspondence and propose StereoBind, a framework that binds visual source motion to stereo sound generation.
- Visual Motion Binding establishes source-aware audiovisual correspondence, the Spatial Track Encoder captures absolute source positions, and Residual Track RoPE models relative motion.
- For supervision and evaluation, we construct StereoWorld-29K, a large-scale stereo audio-video dataset with paired motion tracks, and StereoWorldBench for measuring audiovisual spatial consistency.
- Experiments show that StereoBind substantially improves spatial alignment in stereo audio generation over existing models while preserving overall audiovisual quality.
Sources (1)
- [1]Here the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio GenerationHugging Face Daily Papers · Sep 30, 12:00 AM
Recent joint video-audio generation models have achieved strong semantic correspondence and temporal synchronization.
We refer to this property as Dynamic Spatial Correspondence and propose StereoBind, a framework that binds visual source motion to stereo sound generation.
Extractive summary: sentences quoted from the sources.