Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction

Key points

• We study this question for masked-text reconstruction using a smoothed predictor built from nearby token-pair counts and evaluated over the full 50,257-token GPT-2 vocabulary.1
• Treating each source document as one observation, we derive a first-order formula for the expected disagreement between predictors fitted to two independent corpora.1
• Forecast accuracy depends less on how these correlations are modeled than on the size of the reference corpus: small references systematically underpredict disagreement.1
• Finally, small masked diffusion language models trained on the same corpora show that the count-based forecasts partly identify which inputs are most sensitive to the corpus, but miss the neural models' much faster decline in disagreement.1

Sources (1)

• [1]Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 04:40 AM

  If a second text corpus were collected from the same population, how much would a predictor's outputs change?

  We study this question for masked-text reconstruction using a smoothed predictor built from nearby token-pair counts and evaluated over the full 50,257-token GPT-2 vocabulary.

Extractive summary: sentences quoted from the sources.