Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge.
ProofPaper ↗
Key points
- Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features.
- We introduce Monarch-MoE, which replaces the dense factors in LoRA experts with block-diagonal factors connected by a fixed permutation.
- To make these compact experts practical on memory-limited hardware, we combine batched computation, fused forward operators, and an optimized backward pass.
- Across three continual-learning methods evaluated on MLLM-DCL and UCIT with LLaVA-1.5-7B, Monarch-MoE improves first-task retention while reducing adapter parameters by approximately 75% and 50%, respectively.
Sources (1)
- [1]Monarch-MoE: Structured Experts for Multimodal Continual Instruction TuningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 08:33 AM
Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge.
Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Introducing Clef-omni with full multimodality, plus a faster Clef and a cheaper Clef-flash
- Oct 8, 2026A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges
- Oct 8, 2026Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention
- Oct 8, 2026Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts
- Oct 8, 2026DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
- Oct 7, 2026SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routing
