Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models

Key points

• We show the answer is encoded in the spectral structure of pretrained weights.1
• We prove that the OOD accuracy gap is bounded by how tightly the source representations concentrate.1
• The selection does not leak the target: for each model family outside the matrix we logged the cell, metric and sign before running its OOD evaluation, and the predicted direction held in every case: EEG, genomic and protein.1
• The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time, before data or compute is committed to a target domain.1

Sources (1)

• [1]Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 09:04 AM

  Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available?

  We show the answer is encoded in the spectral structure of pretrained weights.

Extractive summary: sentences quoted from the sources.