Democratizing MoE inference on commodity GPUs with CoMoE

Key points

• Deploying Mixture-of-Experts (MoE) models relies heavily on Expert Parallelism, which generates intense inter-GPU communication.1
• Consequently, state-of-the-art inference systems require high-bandwidth, P2P interconnects (e.g., NVLink) in datacenter GPUs to handle massive token routing, making deployment prohibitively expensive.1
• For token combine, we propose a fine-grained, token-level aggregation mechanism using host staging buffers, which replaces rigid global synchronization and mitigates straggler effects.1
• Evaluation on RTX 5090 GPUs shows that CoMoE improves inference throughput by up to 1.46x, approaching the performance of NVLink-capable A800 GPUs at only 23.4% of the hardware cost.1

Sources (1)

• [1]Democratizing MoE inference on commodity GPUs with CoMoE
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 04:24 AM

  We present CoMoE, a communication-efficient MoE inference system that resolves this mismatch through novel host-centric routing.

  Deploying Mixture-of-Experts (MoE) models relies heavily on Expert Parallelism, which generates intense inter-GPU communication.

Extractive summary: sentences quoted from the sources.