Towards Calibrated Probabilistic Forecasts for Events of Interest via Outcome-Conditional Recalibration

Key points

• Calibration is an essential requirement for probabilistic predictions to be useful for decision making.1
• While state-of-the-art prediction methods often yield miscalibrated predictive distributions, several post-hoc recalibration schemes have been proposed to generate calibrated predictions.1
• It works by applying the quantile recalibration approach of Kuleshov et al. (2018) to forecast conditional distributions, before rescaling these conditional distributions so that forecast event probabilities match empirical occurrence frequencies.1
• Across regression benchmarks, we demonstrate that existing recalibration schemes do not necessarily yield calibrated predictions when interest is on particular outcomes, and that our approach improves outcome-conditional calibration relative to existing conditional and unconditional recalibration methods, while retaining competitive calibration overall.1

Sources (1)

• [1]Towards Calibrated Probabilistic Forecasts for Events of Interest via Outcome-Conditional Recalibration
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 01:41 PM

  Hence, in this paper, we introduce outcome-conditional recalibration, a post-hoc method to recalibrate probabilistic predictions on user-defined regions of the outcome space.

  Calibration is an essential requirement for probabilistic predictions to be useful for decision making.

Extractive summary: sentences quoted from the sources.