Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention

Key points

• We find that fine-tuning different Transformer layers produces different target-task gains and degrees of capability degradation, suggesting that not all layers are equally suitable for adaptation.1
• We therefore introduce input--output cosine similarity as a lightweight, forward-only proxy for ranking layer sensitivity.1
• Building on this observation, we propose Layer-Selective LoRA (LS-LoRA), which places trainable LoRA adapters only in layers with low input--output similarity.1
• Experiments on mathematical reasoning and code generation show that LS-LoRA improves average target-task performance while retaining substantially more commonsense reasoning capability than standard all-layer LoRA, demonstrating that carefully choosing where to adapt can provide a simple and effective way to balance target-task adaptation and general capability retention.1

Sources (1)

• [1]Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 09:59 AM

  Parameter-efficient fine-tuning (PEFT) enables large language models (LLMs) to adapt to specialized tasks, but often at the cost of degrading general capabilities acquired during pretraining.

  We find that fine-tuning different Transformer layers produces different target-task gains and degrees of capability degradation, suggesting that not all layers are equally suitable for adaptation.

Extractive summary: sentences quoted from the sources.