Non-Asymptotic Approximations of Neural Networks by Gaussian Processes
Abstract
We study the extent to which wide neural networks may be approximated by Gaussian processes, when initialized with random weights. It is a well-established fact that as the width of a network goes to infinity, its law converges to that of a Gaussian process. We make this quantitative by establishing explicit convergence rates for the central limit theorem in an infinite-dimensional functional space, metrized with a natural transportation distance. We identify two regimes of interest; when the activation function is polynomial, its degree determines the rate of convergence, while for non-polynomial activations, the rate is governed by the smoothness of the function.
Cite
Text
Eldan et al. "Non-Asymptotic Approximations of Neural Networks by Gaussian Processes." Conference on Learning Theory, 2021.Markdown
[Eldan et al. "Non-Asymptotic Approximations of Neural Networks by Gaussian Processes." Conference on Learning Theory, 2021.](https://mlanthology.org/colt/2021/eldan2021colt-nonasymptotic/)BibTeX
@inproceedings{eldan2021colt-nonasymptotic,
title = {{Non-Asymptotic Approximations of Neural Networks by Gaussian Processes}},
author = {Eldan, Ronen and Mikulincer, Dan and Schramm, Tselil},
booktitle = {Conference on Learning Theory},
year = {2021},
pages = {1754-1775},
volume = {134},
url = {https://mlanthology.org/colt/2021/eldan2021colt-nonasymptotic/}
}