Adaptive Mean Estimation by In-Context Learning: A Gradient-Flow Analysis

Key points

• We study how such adaptivity is learned in a controlled location-estimation problem.1
• We combine attention experts through either a softmax mixture of experts or a gated linear unit (GLU), and analyze stagewise gradient flow.1
• With $\widetildeΩ(n^{1+ε})$ pretraining tasks, the learned estimator is asymptotically efficient on Gaussian tasks, within a factor $n^ε$ of the minimax rate on uniform tasks, and order-optimal on mixtures in a shrinking-variance regime.1
• Softmax gating enforces normalization and exact translation equivariance, whereas the GLU must learn it: its dynamics separate into fast bias removal followed by slow expert selection.1

Sources (1)

• [1]Adaptive Mean Estimation by In-Context Learning: A Gradient-Flow Analysis
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 05:57 AM

  Prior Fitted Networks (PFNs) such as TabPFN now rival established statistical procedures across prediction and estimation tasks.

  We study how such adaptivity is learned in a controlled location-estimation problem.

Extractive summary: sentences quoted from the sources.