PAC-Bayes and Information Complexity
Abstract
We point out that a number of well-known PAC-Bayesian-style and information-theoretic generalization bounds for randomized learning algorithms can be derived under a common framework starting from a fundamental information exponential inequality. We also obtain new bounds for data-dependent priors and unbounded loss functions. Optimizing these bounds naturally gives rise to a method called Information Complexity Minimization for which we discuss two practical examples for learning with neural networks, namely Entropy- and PAC-Bayes- SGD.
Cite
Text
Banerjee and Montufar. "PAC-Bayes and Information Complexity." ICLR 2021 Workshops: Neural_Compression, 2021.Markdown
[Banerjee and Montufar. "PAC-Bayes and Information Complexity." ICLR 2021 Workshops: Neural_Compression, 2021.](https://mlanthology.org/iclrw/2021/banerjee2021iclrw-pacbayes/)BibTeX
@inproceedings{banerjee2021iclrw-pacbayes,
title = {{PAC-Bayes and Information Complexity}},
author = {Banerjee, Pradeep Kr. and Montufar, Guido},
booktitle = {ICLR 2021 Workshops: Neural_Compression},
year = {2021},
url = {https://mlanthology.org/iclrw/2021/banerjee2021iclrw-pacbayes/}
}