Sequential Pretraining Favors Large Models
Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training?
ProofPaper ↗
Key points
- Large neural networks often acquire capabilities that small models fail to learn.
- We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning.
- We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect.
- Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining.
Sources (1)
- [1]Sequential Pretraining Favors Large ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 07:54 AM
Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training?
Large neural networks often acquire capabilities that small models fail to learn.
Extractive summary: sentences quoted from the sources.