Continually Post-training Small Models via Synthetic Self-distillation

Key points

• While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities.1
• We show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift, revealing the necessity of in-distribution learning.1
• We thus introduce Synthetic Self-Distillation (SynthSelf), imitating SDFT's in-distribution rewrite by outsourcing the task to a minimally larger, same-family aligner.1
• Extensive further analyses confirm SynthSelf paves the way toward effective small-model continual learning.1

Sources (2)

• [1]Continually Post-training Small Models via Synthetic Self-distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 07:06 AM

  Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge.

  While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities.

• [2]Continually Post-training Small Models via Synthetic Self-distillation
  Hugging Face Daily Papers · Oct 9, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.