Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings

Key points

• Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs.1
• However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations.1
• Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization.1
• Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.1

Sources (1)

• [1]Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 04:26 PM

  To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration.

  Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs.

Extractive summary: sentences quoted from the sources.