SV-TAD: Native Sparse Convs for Efficient Temporal Action Detection

Key points

• While effective for parameter-efficient training, existing adapters do not reduce inference-time computation, leaving scalability with respect to video length largely unaddressed.1
• Token selection can reduce attention cost by pruning redundant tokens, but it breaks the spatial grid structure required by convolutional adapters.1
• We address this by introducing native sparse 2D convolutions, a primitive that allows these adapters, for the first time, to operate directly and efficiently on dynamically pruned token sets.1
• We integrate this primitive into SV-TAD, an adapter framework for temporal action detection, reducing VideoMAEv2-L computation by up to 64% and achieving 2.2x faster inference, while maintaining state-of-the-art accuracy on THUMOS-14 and ActivityNet-1.3.1

Sources (1)

• [1]SV-TAD: Native Sparse Convs for Efficient Temporal Action Detection
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 09:31 AM

  To adapt billion-parameter Vision Transformers for long-video understanding, recent methods freeze the backbone and train lightweight convolutional modules.

  While effective for parameter-efficient training, existing adapters do not reduce inference-time computation, leaving scalability with respect to video length largely unaddressed.

Extractive summary: sentences quoted from the sources.