Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
Abstract
We perform scalable approximate inference in continuous-depth Bayesian neural networks. In this model class, uncertainty about separate weights in each layer gives hidden units that follow a stochastic differential equation. We demonstrate gradient-based stochastic variational inference in this infinite-parameter setting, producing arbitrarily-flexible approximate posteriors. We also derive a novel gradient estimator that approaches zero variance as the approximate posterior over weights approaches the true posterior. This approach brings continuous-depth Bayesian neural nets to a competitive comparison against discrete-depth alternatives, while inheriting the memory-efficient training and tunable precision of Neural ODEs.
Cite
Text
Xu et al. "Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations." Artificial Intelligence and Statistics, 2022.Markdown
[Xu et al. "Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations." Artificial Intelligence and Statistics, 2022.](https://mlanthology.org/aistats/2022/xu2022aistats-infinitely/)BibTeX
@inproceedings{xu2022aistats-infinitely,
title = {{Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations}},
author = {Xu, Winnie and Chen, Ricky T. Q. and Li, Xuechen and Duvenaud, David},
booktitle = {Artificial Intelligence and Statistics},
year = {2022},
pages = {721-738},
volume = {151},
url = {https://mlanthology.org/aistats/2022/xu2022aistats-infinitely/}
}