Overcoming Data Imbalances with Expert Selection Biases

Key points

• However, when data is imbalanced across categories, minority categories can become concentrated in a small fraction of the expert pool or spread inefficiently across them, effectively limiting their available capacity.1
• In this work, we demonstrate that category-conditioned expert-selection biases can preferentially route tokens from different categories toward specialized expert pools.1
• By applying biases based on input category, we provide a simple way to induce soft category-aligned specialization.1
• Overall, our results show that expert biases can serve as a lightweight control for category-level expert allocation, making room for low-resource categories while retaining standard MoE inference.1

Sources (1)

• [1]Overcoming Data Imbalances with Expert Selection Biases
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 10:45 AM

  Mixture-of-Experts (MoE) models have become the standard architecture for frontier LLMs. The multiple parallel experts in each transformer block theoretically enable expert specialization according to categories such as domains, languages, or modalities during pretraining.

  However, when data is imbalanced across categories, minority categories can become concentrated in a small fraction of the expert pool or spread inefficiently across them, effectively limiting their available capacity.

Extractive summary: sentences quoted from the sources.