The Asymptotic Spectrum of the Hessian of DNN Throughout Training
Abstract
The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs: we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training.
Cite
Text
Jacot et al. "The Asymptotic Spectrum of the Hessian of DNN Throughout Training." International Conference on Learning Representations, 2020.Markdown
[Jacot et al. "The Asymptotic Spectrum of the Hessian of DNN Throughout Training." International Conference on Learning Representations, 2020.](https://mlanthology.org/iclr/2020/jacot2020iclr-asymptotic/)BibTeX
@inproceedings{jacot2020iclr-asymptotic,
title = {{The Asymptotic Spectrum of the Hessian of DNN Throughout Training}},
author = {Jacot, Arthur and Gabriel, Franck and Hongler, Clément},
booktitle = {International Conference on Learning Representations},
year = {2020},
url = {https://mlanthology.org/iclr/2020/jacot2020iclr-asymptotic/}
}