MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos

Key points

• We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case.1
• We introduce MacJEPA, a missing-modality-robust Masked-context query JEPA that recognizes visual actions and acoustic events from supplied interval queries over audio-visual context.1
• MacJEPA further repurposes masking in JEPA from a self-supervised pretext into a supervised robustness objective, aligning masked and clean latent representations of both multimodal content tokens and the task-conditioned queries.1
• MacJEPA thus unifies strong full-input recognition with temporal missing-modality robustness in a single model operating on untrimmed multi-event videos.1

Sources (1)

• [1]MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 11:42 AM

  Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at inference.

  We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case.

Extractive summary: sentences quoted from the sources.