Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression

Key points

• We analyze three MoE compression families: expert pruning, expert merging, and weight reconstruction, and derive structural error bounds showing that pruning and merging can incur non-vanishing errors tied to routing and expert heterogeneity.1
• In contrast, weight reconstruction avoids these structural costs by preserving expert structure and routing.1
• Motivated by the analysis, we propose Shared Low-rank Basis Factorization (SLBF), a data-free weight reconstruction method that uses rank-$k$ bases shared among experts, enabling richer cross-expert sharing, faster convergence, and lower reconstruction error.1
• Across five MoE architectures spanning 16B to 122B parameters, SLBF consistently outperforms methods from all three compression families.1

Sources (1)

• [1]Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 03:01 AM

  Mixture-of-Experts (MoE) large language models decouple capacity from compute through sparse routing, but their large parameter count creates storage and serving challenges.

  We analyze three MoE compression families: expert pruning, expert merging, and weight reconstruction, and derive structural error bounds showing that pruning and merging can incur non-vanishing errors tied to routing and expert heterogeneity.

Extractive summary: sentences quoted from the sources.