OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

Key points

• Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits.1
• Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability.1
• OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons.1
• Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio--visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development.1

Sources (2)

• [1]OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning
  Hugging Face Daily Papers · Oct 8, 12:00 AM

  We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework.

  Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits.

• [2]OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:59 PM · same content

Extractive summary: sentences quoted from the sources.