GIST: Generator-state-Informed Supervision for Tabular Foundation Models

Key points

• Tabular foundation models (TFMs) based on prior-data fitted networks typically use sampled hard query labels, leaving avoidable sampling variability in pretraining.1
• GIST jointly designs task generation and query supervision, using entropy-calibrated encodings to obtain tractable, input-consistent targets on supported label-dependent branches.1
• We show that this averaging preserves the population objective and yields no larger gradient covariance at fixed parameters within each generating distribution.1
• Under matched pretraining schedules, GIST improves mean rank across three architectures on both TabArena and TALENT without changing architecture or inference.1

Sources (1)

• [1]GIST: Generator-state-Informed Supervision for Tabular Foundation Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:52 AM

  We propose Generator-state-Informed Supervision for TFMs (GIST), which uses these states to average query losses over conditional label distributions.

  Tabular foundation models (TFMs) based on prior-data fitted networks typically use sampled hard query labels, leaving avoidable sampling variability in pretraining.

Extractive summary: sentences quoted from the sources.