A Differential Equation for Modeling Nesterov’s Accelerated Gradient Method: Theory and Insights
Abstract
We derive a second-order ordinary differential equation (ODE), which is the limit of Nesterov’s accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov’s scheme and thus can serve as a tool for analysis. We show that the continuous time ODE allows for a better understanding of Nesterov’s scheme. As a byproduct, we obtain a family of schemes with similar convergence rates. The ODE interpretation also suggests restarting Nesterov’s scheme leading to an algorithm, which can be rigorously proven to converge at a linear rate whenever the objective is strongly convex.
Cite
Text
Su et al. "A Differential Equation for Modeling Nesterov’s Accelerated Gradient Method: Theory and Insights." Neural Information Processing Systems, 2014.Markdown
[Su et al. "A Differential Equation for Modeling Nesterov’s Accelerated Gradient Method: Theory and Insights." Neural Information Processing Systems, 2014.](https://mlanthology.org/neurips/2014/su2014neurips-differential/)BibTeX
@inproceedings{su2014neurips-differential,
title = {{A Differential Equation for Modeling Nesterov’s Accelerated Gradient Method: Theory and Insights}},
author = {Su, Weijie and Boyd, Stephen and Candes, Emmanuel},
booktitle = {Neural Information Processing Systems},
year = {2014},
pages = {2510-2518},
url = {https://mlanthology.org/neurips/2014/su2014neurips-differential/}
}