Prospective Prediction of OOD Degradation from Source-Side Training Dynamics
We study whether persistent out-of-distribution (OOD) degradation can be predicted before it is directly observed using only source-side training dynamics.
ProofPaper ↗
Key points
- In a controlled shortcut-learning setting, a simple logistic regression predictor develops a clear prospective signal, while training time alone does not.
- Temporal summaries of the source-side quantities are substantially more informative than their current values.
- When transferred without additional training from a CNN to an MLP, confidence and entropy dynamics retain substantial predictive information.
- These results provide a proof of principle that source-side training dynamics can contain an early warning signal for future OOD failure.
Sources (1)
- [1]Prospective Prediction of OOD Degradation from Source-Side Training DynamicsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:41 PM
We study whether persistent out-of-distribution (OOD) degradation can be predicted before it is directly observed using only source-side training dynamics.
In a controlled shortcut-learning setting, a simple logistic regression predictor develops a clear prospective signal, while training time alone does not.
Extractive summary: sentences quoted from the sources.