MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation

Key points

• Given any subset of modalities, MUNITE models the conditional distribution over the latent representation associated with the complete observation.1
• A shared latent sample captures variation that must remain consistent across generated targets, while modality-specific generative decoders model the remaining uncertainty independently.1
• To learn these conditional distributions from incomplete training examples, we extend conditional flow matching through self-distillation: predictions conditioned on richer available observations supervise the same model conditioned on smaller subsets at the same intermediate latent state.1
• Across PolyMNIST-D-Q, FFHQ64, and image-text-audio, MUNITE achieves competitive or better generation quality and source-target alignment, with higher joint-generation coherence.1

Sources (1)

• [1]MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 11:27 AM

  We introduce MUNITE, a latent-variable framework for flexible any-to-any multimodal generation that treats encoding and latent generation as the same inference problem under different amounts of observed evidence.

  Given any subset of modalities, MUNITE models the conditional distribution over the latent representation associated with the complete observation.

Extractive summary: sentences quoted from the sources.