The optimal information complexity of VC learning
Steinke and Zakynthinou(2020) introduces the Conditional Mutual Information (CMI) framework of analyzing the information complexity of learning algorithms based on algorithm-dependent information-theoretic quantities.
ProofPaper ↗
Key points
- We study one of these quantities, the evaluated Conditional Mutual Information (eCMI).
- It has been an interesting question whether the optimal PAC guarantee for VC classes can be recovered from the algorithm-dependent analyses via CMI.
- And we show that it is possible to recover this guarantee by constructing a learning algorithm whose eCMI is of order O(d) in the realizable case, where d is the VC-dimension of the concept class.
- Specially, our algorithm is a randomized Majority-of-5 base learners with optimal in-expectation generalization guarantee.
Sources (1)
- [1]The optimal information complexity of VC learningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 10:58 PM
Steinke and Zakynthinou(2020) introduces the Conditional Mutual Information (CMI) framework of analyzing the information complexity of learning algorithms based on algorithm-dependent information-theoretic quantities.
We study one of these quantities, the evaluated Conditional Mutual Information (eCMI).
Extractive summary: sentences quoted from the sources.