Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent
Abstract
We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate - infinite width scaling regime.
Cite
Text
Gess et al. "Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent." Journal of Machine Learning Research, 2024.Markdown
[Gess et al. "Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent." Journal of Machine Learning Research, 2024.](https://mlanthology.org/jmlr/2024/gess2024jmlr-stochastic/)BibTeX
@article{gess2024jmlr-stochastic,
title = {{Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent}},
author = {Gess, Benjamin and Kassing, Sebastian and Konarovskyi, Vitalii},
journal = {Journal of Machine Learning Research},
year = {2024},
pages = {1-27},
volume = {25},
url = {https://mlanthology.org/jmlr/2024/gess2024jmlr-stochastic/}
}