Prediction Is Not a Sufficient Memory State

Key points

• We characterize this distinction for exact cumulative learning and attention.1
• For smooth convex residual losses with unrestricted real features and labels, a continuous recurrent state of fixed dimension exists exactly when the loss is polynomial.1
• With ordinary, unweighted observations, degree $D$ requires precisely $r=\binom{d+D}{D}-1$ coordinates; even among histories with one fixed current predictor, exactly $r-d$ coordinates can remain necessary for continuation.1
• For ridge regression, the hidden evidence is the design matrix: the exact state costs $dp+d(d+1)/2$, versus $dp$ for the predictor.1

Sources (1)

• [1]Prediction Is Not a Sufficient Memory State
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 05:27 PM

  A memory that reproduces every current prediction may still have forgotten how to learn from the next observation.

  We characterize this distinction for exact cumulative learning and attention.

Extractive summary: sentences quoted from the sources.