Tabular Foundation Models Are Effectively Shallow

Key points

• Meanwhile, traditional alternatives like Gradient-Boosted Decision Trees (GBDTs) remain highly competitive and run efficiently on CPUs, suggesting that tabular tasks may not inherently require deep architectures.1
• This contrast raises a fundamental question: can TFMs be simplified by removing architectural redundancies without sacrificing performance?1
• To answer this, we introduce a closed-form, label-free affine translator that replaces a contiguous window of up to 96% of the blocks, leaving the rest unchanged.1
• Evaluating eight frontier TFMs on 171 classification datasets, we reveal that tabular transformers are vastly overparameterized: replacing half of a model's blocks costs at most $0.002$ median AUROC across all architectures, with the half-depth TabFM still outperforming every other full-depth TFMs and tuned GBDTs.1

Sources (1)

• [1]Tabular Foundation Models Are Effectively Shallow
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 09:01 AM

  In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without additional training.

  Meanwhile, traditional alternatives like Gradient-Boosted Decision Trees (GBDTs) remain highly competitive and run efficiently on CPUs, suggesting that tabular tasks may not inherently require deep architectures.

Extractive summary: sentences quoted from the sources.