Dissecting Non-Vacuous Generalization Bounds Based on the Mean-Field Approximation
Abstract
Explaining how overparametrized neural networks simultaneously achieve low risk and zero empirical risk on benchmark datasets is an open problem. PAC-Bayes bounds optimized using variational inference (VI) have been recently proposed as a promising direction in obtaining non-vacuous bounds. We show empirically that this approach gives negligible gains when modelling the posterior as a Gaussian with diagonal covariance—known as the mean-field approximation. We investigate common explanations, such as the failure of VI due to problems in optimization or choosing a suboptimal prior. Our results suggest that investigating richer posteriors is the most promising direction forward.
Cite
Text
Pitas. "Dissecting Non-Vacuous Generalization Bounds Based on the Mean-Field Approximation." International Conference on Machine Learning, 2020.Markdown
[Pitas. "Dissecting Non-Vacuous Generalization Bounds Based on the Mean-Field Approximation." International Conference on Machine Learning, 2020.](https://mlanthology.org/icml/2020/pitas2020icml-dissecting/)BibTeX
@inproceedings{pitas2020icml-dissecting,
title = {{Dissecting Non-Vacuous Generalization Bounds Based on the Mean-Field Approximation}},
author = {Pitas, Konstantinos},
booktitle = {International Conference on Machine Learning},
year = {2020},
pages = {7739-7749},
volume = {119},
url = {https://mlanthology.org/icml/2020/pitas2020icml-dissecting/}
}