An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information

Key points

• Accuracy is known to force information into the model; data processing does not carry such lower bounds to losses.1
• We show, by bounding three moments of the loss differences, that accuracy also forces ld-CMI.1
• Consequently, range-scaled ld-CMI bounds cannot vanish on these distributions, although every proper learner's generalization gap is $O(n^{-1/2})$.1
• We also show that model-level information does not determine noisy loss-difference information, and that the growth, slope and dimension conditions are needed, the last up to a logarithm.1

Sources (1)

• [1]An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 11:07 PM

  Loss-difference conditional mutual information (ld-CMI) uses the smallest of the standard observations in the supersample hierarchy of generalization bounds: it measures what a learner's loss differences reveal about which candidate of each pair it was trained on.

  Accuracy is known to force information into the model; data processing does not carry such lower bounds to losses.

Extractive summary: sentences quoted from the sources.