MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling

Key points

• Sparsely activated Mixture-of-Experts (MoE) models increase model capacity without a proportional increase in per-token computation.1
• Dense-to-MoE upcycling reuses pretrained dense models to construct such systems, commonly by copying feed-forward networks (FFNs) into independently trained experts.1
• Each expert is defined by a learned binary mask, and a token-level router selects which masked FFNs to execute and combine.1
• These results establish mask learning over frozen weights as a practical alternative for constructing token-routed MoE experts.1

Sources (1)

• [1]MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 06:04 AM

  We introduce MASKerade, a dense-to-MoE training method that instead learns experts as sparse subnetworks of a frozen pretrained FFN.

  Sparsely activated Mixture-of-Experts (MoE) models increase model capacity without a proportional increase in per-token computation.

Extractive summary: sentences quoted from the sources.