TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy

Key points

• We assess reliability through point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency.1
• We then introduce TabReliBench, a large classification and regression benchmark for these comparisons and dataset-level analyses.1
• We find that TFMs' rankings change between TabArena and our benchmark and across reliability metrics.1
• Ensembling improves predictive performance but does not consistently improve calibration of PPDs; TFMs also benefit little from conformal post-processing, which does not reliably resolve subgroup gaps.1

Sources (1)

• [1]TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:29 PM

  Tabular foundation models (TFMs) are increasingly popular for their strong predictive performance.

  We assess reliability through point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency.

Extractive summary: sentences quoted from the sources.