Thinking in Depth: Retrospective Inference for Tabular Foundation Models

Key points

• Tabular foundation models (TFMs) are pretrained across diverse tabular tasks and make predictions on a new table at inference time using its labeled examples as context.1
• By tracing individual queries through several strong TFMs, we find that predictive refinement is highly uneven across depth and is often concentrated in later layers.1
• Attention Residuals address the former by adaptively reweighting contributions from different depths, while query-conditioned Gated Attention addresses the latter by modulating the attention output element-wise across representation dimensions.1
• Our analysis shows that Retro shifts predictive refinement earlier and more broadly across depth, with different stages revising different subsets of queries in a pattern suggestive of multi-view refinement.1

Sources (1)

• [1]Thinking in Depth: Retrospective Inference for Tabular Foundation Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 04:11 PM

  We introduce Retro, a tabular foundation model based on retrospective inference, where later stages can explicitly revisit and recombine intermediate information produced earlier in the network.

  Tabular foundation models (TFMs) are pretrained across diverse tabular tasks and make predictions on a new table at inference time using its labeled examples as context.

Extractive summary: sentences quoted from the sources.