Improving the Gaussian Approximation in Neural Networks: Para-Gaussians and Edgeworth Expansions
Abstract
Gaussian approximations are often used for developing the theory of how neural networks scale as the number of neurons grows large. However, it is known that these approximations break down as depth increases due to the accumulation of approximation errors. To remedy this, we provide a new family of distributions that appear naturally in neural networks and provide more accurate approximations than the usual Gaussian approximation. We develop a method for obtaining the probability density function via Hermite polynomials and connect this to the classical Edgeworth expansion.
Cite
Text
Nica and Ortmann. "Improving the Gaussian Approximation in Neural Networks: Para-Gaussians and Edgeworth Expansions." NeurIPS 2024 Workshops: M3L, 2024.Markdown
[Nica and Ortmann. "Improving the Gaussian Approximation in Neural Networks: Para-Gaussians and Edgeworth Expansions." NeurIPS 2024 Workshops: M3L, 2024.](https://mlanthology.org/neuripsw/2024/nica2024neuripsw-improving/)BibTeX
@inproceedings{nica2024neuripsw-improving,
title = {{Improving the Gaussian Approximation in Neural Networks: Para-Gaussians and Edgeworth Expansions}},
author = {Nica, Mihai and Ortmann, Janosch},
booktitle = {NeurIPS 2024 Workshops: M3L},
year = {2024},
url = {https://mlanthology.org/neuripsw/2024/nica2024neuripsw-improving/}
}