Learning by Compression: Risk Estimates Attain the Law-of-Large-Numbers Rate

Key points

• For learning rules based on preferent compression, existing results upper-bound the risk in terms of a sample-dependent complexity $\boldsymbol{k}$, defined as the cardinality of the compression augmented by the training examples on which the reconstructed hypothesis is inappropriate.1
• We prove that, for every fixed confidence level $1-δ$, the risk is bounded with probability at least $1-δ$ by $$ \frac{\boldsymbol{k}}{N} + O\left(\frac{1}{\sqrt{N}}\right), $$ uniformly over all possible values of $\boldsymbol{k}$.1
• Thus, risk estimates for learning algorithms based on preferent compression attain the same law-of-large-numbers rate as the empirical estimate of the probability of a fixed event.1
• In particular, we identify regimes in which the empirical risk retains the $O(1/\sqrt{N})$ convergence rate, as well as regimes in which it converges more slowly or ceases to be consistent, while the complexity-based estimator preserves the $O(1/\sqrt{N})$ rate.1

Sources (1)

• [1]Learning by Compression: Risk Estimates Attain the Law-of-Large-Numbers Rate
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 11:44 AM

  Compression schemes provide a general framework for deriving data-dependent risk bounds for learning algorithms.

  For learning rules based on preferent compression, existing results upper-bound the risk in terms of a sample-dependent complexity $\boldsymbol{k}$, defined as the cardinality of the compression augmented by the training examples on which the reconstructed hypothesis is inappropriate.

Extractive summary: sentences quoted from the sources.