X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation

Key points

• To tackle these issues, we propose X-AVDD, a cross-attention-based distillation method that explicitly captures inter-modal relationships during synthesis.1
• By decoupling cross-modal alignment from joint optimization, X-AVDD is substantially more memory-efficient, reducing end-to-end synthesis cost by $\sim 8\times$ GPU-hours, while producing highly aligned synthetic data.1
• On VGGSound, an audio-visual dataset of short YouTube clips, X-AVDD improves top-1 test accuracy from $8.2%\rightarrow24.0%$ at SPC=$10$ and $9.8%\rightarrow26.5%$ at SPC=$20$.1
• Furthermore, X-AVDD improves cross-architecture generalization (e.g., $25.5%\rightarrow35.3%$ for a ViT target on AVE dataset at SPC $=50$) and successfully scales to complex architectures such as AudioCLIP, achieving $62.3%$ accuracy using only $5.65%$ of the full dataset, against the full dataset accuracy of $72.4%$ while completing in $9\times$ lower training time compared to the full dataset.1

Sources (1)

• [1]X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:51 PM

  Audio-visual dataset distillation aims to synthesize a small dataset from a large paired audio-visual dataset, while nearly preserving the training utility for neural networks.

  To tackle these issues, we propose X-AVDD, a cross-attention-based distillation method that explicitly captures inter-modal relationships during synthesis.

Extractive summary: sentences quoted from the sources.