Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks
Abstract
Both PAC-Bayesian and Sample Compress learning frameworks have been shown instrumental for deriving tight (non-vacuous) generalization bounds for neural networks. We leverage these results in a meta-learning scheme, relying on a hypernetwork that outputs the parameters of a downstream predictor from a dataset input. The originality of our approach lies in the investigated hypernetwork architectures that encode the dataset before decoding the parameters: (1) a PAC-Bayesian encoder that expresses a posterior distribution over a latent space, (2) a Sample Compress encoder that selects a small sample of the dataset input along with a message from a discrete set, and (3) a hybrid between both approaches motivated by a new Sample Compress theorem handling continuous messages. The latter theorem exploits the pivotal information transiting at the encoder-decoder junction in order to compute generalization guarantees for each downstream predictor obtained by our meta-learning scheme.
Cite
Text
Leblanc et al. "Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks." Proceedings of the 42nd International Conference on Machine Learning, 2025.Markdown
[Leblanc et al. "Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks." Proceedings of the 42nd International Conference on Machine Learning, 2025.](https://mlanthology.org/icml/2025/leblanc2025icml-generalization/)BibTeX
@inproceedings{leblanc2025icml-generalization,
title = {{Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks}},
author = {Leblanc, Benjamin and Bazinet, Mathieu and D’Amours, Nathaniel and Drouin, Alexandre and Germain, Pascal},
booktitle = {Proceedings of the 42nd International Conference on Machine Learning},
year = {2025},
pages = {32830-32852},
volume = {267},
url = {https://mlanthology.org/icml/2025/leblanc2025icml-generalization/}
}