Conformal Prediction Sets Quantify Information Gain: A Theoretical Perspective

Key points

• In this work, we provide such a foundation using a decision-theoretic generalization of entropy tailored to set-valued prediction.1
• In particular, we introduce a family of generalized information measures based on the size and coverage of conformal prediction sets.1
• Together, our results formally relate conformal prediction to classical information-theoretic quantities and justify using set-size reduction as an information gain metric.1
• Empirically, we validate our theory across 11 classification settings and show that set-size reduction and Shannon mutual information can rank features differently in a greedy feature selection experiment.1

Sources (1)

• [1]Conformal Prediction Sets Quantify Information Gain: A Theoretical Perspective
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 05:59 PM

  Conformal prediction is a popular tool for uncertainty quantification that outputs prediction sets with finite-sample coverage guarantees.

  In this work, we provide such a foundation using a decision-theoretic generalization of entropy tailored to set-valued prediction.

Extractive summary: sentences quoted from the sources.