RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing

Key points

• Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active for processing a fraction of tokens.1
• We propose and present evidence for an alternative account that we call the Superposed Specialisation Hypothesis (SSH): experts specialise in a disjoint union of fine-grained features rather than one broad domain.1
• On gpt-oss-20b, RouterInterp explains expert routing with ${\sim}65%$ higher detection accuracy than prior token statistics based methods.1
• This work provides a scalable method for generating more accurate explanations of expert routing and increases our understanding of a previously uninterpretable component of foundation models.1

Sources (1)

• [1]RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 11:55 AM

  Leveraging the SSH, we introduce RouterInterp, a method for interpreting expert routing that identifies Sparse Autoencoder features most predictive of routing decisions and produces unified natural language explanations.

  Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active for processing a fraction of tokens.

Extractive summary: sentences quoted from the sources.