The Standardization Trap: Certifying Joint Label Processing in Tabular Foundation Models

Key points

• Testing this account using derivatives runs into a standardization trap: public TFM packages standardize the labels before the model sees them, yet ordinary derivatives also reflect behavior outside the set of standardized labels, making a model appear nonlinear even when every prediction it makes agrees with a fixed-weight map.1
• We propose two certificates that depend only on predictions at standardized labels and can reject two distinct explanations: fixed-weight prediction and sums of independent nonlinear label transformations.1
• Across the five public TFMs that we evaluate, our certificates show that changing one context label alters how other labels influence the prediction, a behavior we call joint processing.1
• We further find that joint processing emerges with training and that attention scores carry most of the measured interaction.1

Sources (1)

• [1]The Standardization Trap: Certifying Joint Label Processing in Tabular Foundation Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 01:18 PM

  Linear regression and kernel smoothing offer tractable explanations of in-context learning: in both, the features determine the weight assigned to each context label.

  Testing this account using derivatives runs into a standardization trap: public TFM packages standardize the labels before the model sees them, yet ordinary derivatives also reflect behavior outside the set of standardized labels, making a model appear nonlinear even when every prediction it makes agrees with a fixed-weight map.

Extractive summary: sentences quoted from the sources.