$α$Transfer: Coefficient Transfer for Efficient Model Merging

Key points

• We show that, within the same model family, models exhibit highly congruent performance distributions over merging coefficients across different model sizes.1
• This distributional similarity enables a practical paradigm we call $α$Transfer: searching for optimal coefficients on a small proxy model, then directly transfer them to larger target models.1
• Experimental results demonstrate a 6$\times$ speedup and 70% memory reduction on vision transformers, and a 20$\times$ speedup and 85% memory reduction on large language models, while maintaining comparable performance.1
• Our findings establish $α$Transfer as an efficient and generalizable approach to scaling model merging.1

Sources (1)

• [1]$α$Transfer: Coefficient Transfer for Efficient Model Merging
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 06:13 AM

  Model merging offers a promising solution for combining multiple fine-tuned checkpoints into a single model through parameter arithmetic.

  We show that, within the same model family, models exhibit highly congruent performance distributions over merging coefficients across different model sizes.

Extractive summary: sentences quoted from the sources.