Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix

Key points

• Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data.1
• We identify a parameter-pool confound in comparisons between initialization-time gradient masks and unrestricted random masks under standard zero-initialized LoRA ($B=0$): the gradient with respect to $A$ is identically zero ($\partial\mathcal{L}/\partial A=0$), so gradient masks select strictly from the $B$ matrix whereas unrestricted random masks draw from both $A$ and $B$.1
• Isolating the selection criterion by restricting both approaches to the same parameter pool, we find a negative result: a uniform random mask consistently achieves lower loss-based backward transfer than gradient attribution across seven seeds on Qwen2.5-3B.1
• This advantage is consistent with a stability--plasticity trade-off under the tested task sequence, in which random masking slightly reduces single-task fitting.1

Sources (1)

• [1]Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 09:44 PM

  Parameter-efficient continual learning updates a restricted subset of adapter parameters to incorporate sequential tasks while limiting catastrophic forgetting.

  Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data.

Extractive summary: sentences quoted from the sources.