Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation
Referring Audio--Visual Segmentation (Ref-AVS) is an emergent multimodal task requiring the precise identification and segmentation of a specific object based on a natural language expression grounded in both visual and auditory cues.
ProofPaper ↗
Key points
- Unlike traditional AVS, which segments generic sound sources, or referring video object segmentation (Ref-VOS), which relies solely on visual--textual grounding, Ref-AVS necessitates fine-grained tri-modal interaction across vision, audio, and language.
- To address this, we propose a novel Shift-Aware Identity-guided Latent (SAIL) refinement framework.
- It introduces a dual-token strategy that decouples frame-level dynamics from a global semantic anchor to explicitly model target-shift cues.
- Finally, we leverage a decoder that integrates the refined embeddings with a continuous memory update to maintain spatio-temporal coherence and identity consistency throughout the video sequence.
Sources (1)
- [1]Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual SegmentationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 10:39 PM
Referring Audio--Visual Segmentation (Ref-AVS) is an emergent multimodal task requiring the precise identification and segmentation of a specific object based on a natural language expression grounded in both visual and auditory cues.
Unlike traditional AVS, which segments generic sound sources, or referring video object segmentation (Ref-VOS), which relies solely on visual--textual grounding, Ref-AVS necessitates fine-grained tri-modal interaction across vision, audio, and language.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 8, 2026StereoEngine: Scaling Stereo Bimanual Robot Data without Stereo Cameras
- Oct 8, 2026Visual-Wi-Fi Navigation to Signal-Emitting Devices
- Oct 8, 2026MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
- Oct 8, 2026STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules
- Oct 8, 2026Contextual Correction of Parallel Draftsfor Speculative Decoding
- Oct 8, 2026Reflex: Visually Grounded Reactive Humanoid Control