Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling

Key points

• Existing research has attempted to enhance CLIP's visual representations by incorporating geometric priors from vision-centric models.1
• To address these limitations, we propose SALM, an unsupervised embedding alignment framework based on structurally-aware latent mask modeling.1
• First, we introduce a dual-matrix alignment strategy that explicitly calibrates intra-sample spatial correlations and activation intensities, thereby effectively injecting local geometric priors.1
• Based on this, we further design a latent mask modeling mechanism to guide CLIP to restore the missing semantic details of the target model, thereby implicitly aggregating fine-grained structures into the global semantic space.1

Sources (1)

• [1]Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 03:25 AM

  Vision-Language Models (VLMs) such as CLIP excel in global semantic alignment but often lack fine-grained perceptual capabilities.

  Existing research has attempted to enhance CLIP's visual representations by incorporating geometric priors from vision-centric models.

Extractive summary: sentences quoted from the sources.