Personalized Federated Direct Preference Optimization via LoRA Adapters

Key points

• However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice.1
• We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model.1
• We prove that this decomposition replaces the gradient diversity $κ^2$ in the FedDPO bound with a residual heterogeneity $κ^2{eff}$, which is no larger than $κ^2$ when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge.1
• Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.1

Sources (1)

• [1]Personalized Federated Direct Preference Optimization via LoRA Adapters
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:20 AM

  Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data.

  However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice.

Extractive summary: sentences quoted from the sources.