Estimating Optimal PAC-Bayes Bounds with Hamiltonian Monte Carlo
Abstract
An important yet underexplored question in the PAC-Bayes literature is how much tightness we lose by restricting the posterior family to factorized Gaussian distributions when optimizing a PAC-Bayes bound. We investigate this issue by estimating data-independent PAC-Bayes bounds using the optimal posteriors, comparing them to bounds obtained using MFVI. Concretely, we (1) sample from the optimal Gibbs posterior using Hamiltonian Monte Carlo, (2) estimate its KL divergence from the prior with thermodynamic integration, and (3) propose three methods to obtain high-probability bounds under different assumptions. Our experiments on the MNIST dataset reveal significant tightness gaps, as much as 5-6% in some cases.
Cite
Text
Ujváry et al. "Estimating Optimal PAC-Bayes Bounds with Hamiltonian Monte Carlo." NeurIPS 2023 Workshops: M3L, 2023.Markdown
[Ujváry et al. "Estimating Optimal PAC-Bayes Bounds with Hamiltonian Monte Carlo." NeurIPS 2023 Workshops: M3L, 2023.](https://mlanthology.org/neuripsw/2023/ujvary2023neuripsw-estimating/)BibTeX
@inproceedings{ujvary2023neuripsw-estimating,
title = {{Estimating Optimal PAC-Bayes Bounds with Hamiltonian Monte Carlo}},
author = {Ujváry, Szilvia and Flamich, Gergely and Fortuin, Vincent and Hernández-Lobato, José Miguel},
booktitle = {NeurIPS 2023 Workshops: M3L},
year = {2023},
url = {https://mlanthology.org/neuripsw/2023/ujvary2023neuripsw-estimating/}
}