Asymptotics of Wide Networks from Feynman Diagrams
Abstract
Understanding the asymptotic behavior of wide networks is of considerable interest. In this work, we present a general method for analyzing this large width behavior. The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals. We apply our method to study training dynamics, improving existing bounds and deriving new results on wide network evolution during stochastic gradient descent. Going beyond the strict large width limit, we present closed-form expressions for higher-order terms governing wide network training, and test these predictions empirically.
Cite
Text
Dyer and Gur-Ari. "Asymptotics of Wide Networks from Feynman Diagrams." International Conference on Learning Representations, 2020.Markdown
[Dyer and Gur-Ari. "Asymptotics of Wide Networks from Feynman Diagrams." International Conference on Learning Representations, 2020.](https://mlanthology.org/iclr/2020/dyer2020iclr-asymptotics/)BibTeX
@inproceedings{dyer2020iclr-asymptotics,
title = {{Asymptotics of Wide Networks from Feynman Diagrams}},
author = {Dyer, Ethan and Gur-Ari, Guy},
booktitle = {International Conference on Learning Representations},
year = {2020},
url = {https://mlanthology.org/iclr/2020/dyer2020iclr-asymptotics/}
}