BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models

Key points

• Federated vision-language models (FedVLMs) represent one of the emerging frontiers of machine learning, aiming to integrate federated learning within the fine-tuning pipelines of vision-language models (VLMs).1
• BLEND further introduces a novel loss function that blends the outputs of the global and personalized adapters to achieve personalization, while simultaneously promoting generalization through a principled anchor design implemented via the projection heads of the zero-shot VLM.1
• Extensive experiments demonstrate that BLEND consistently outperforms state-of-the-art baselines, highlighting its ability to effectively balance personalization and generalization across diverse system settings.1

Sources (1)

• [1]BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:57 AM

  In this work, we introduce BLEND, a new framework designed to jointly enhance the personalization and generalization capabilities of FedVLMs. BLEND proposes a selective VLM parameter fine-tuning and aggregation strategy that consists of (i) global vision and text adapters shared across clients, and (ii) a personalized vision projection adapter tailored to each client.

  Federated vision-language models (FedVLMs) represent one of the emerging frontiers of machine learning, aiming to integrate federated learning within the fine-tuning pipelines of vision-language models (VLMs).

Extractive summary: sentences quoted from the sources.