CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

Key points

• Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks.1
• To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters.1
• However, we reveal that such strict geometric constraints trigger an "Orthogonality Dilemma": rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks.1
• To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation.1

Sources (1)

• [1]CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 01:17 PM

  In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge.

  Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks.

Extractive summary: sentences quoted from the sources.