A Stochastic Gradient Method with an Exponential Convergence _Rate for Finite Training Sets
Abstract
We propose a new stochastic gradient method for optimizing the sum of a finite set of smooth functions, where the sum is strongly convex. While standard stochastic gradient methods converge at sublinear rates for this problem, the proposed method incorporates a memory of previous gradient values in order to achieve a linear convergence rate. In a machine learning context, numerical experiments indicate that the new algorithm can dramatically outperform standard algorithms, both in terms of optimizing the training error and reducing the test error quickly.
Cite
Text
Roux et al. "A Stochastic Gradient Method with an Exponential Convergence _Rate for Finite Training Sets." Neural Information Processing Systems, 2012.Markdown
[Roux et al. "A Stochastic Gradient Method with an Exponential Convergence _Rate for Finite Training Sets." Neural Information Processing Systems, 2012.](https://mlanthology.org/neurips/2012/roux2012neurips-stochastic/)BibTeX
@inproceedings{roux2012neurips-stochastic,
title = {{A Stochastic Gradient Method with an Exponential Convergence _Rate for Finite Training Sets}},
author = {Roux, Nicolas L. and Schmidt, Mark and Bach, Francis R.},
booktitle = {Neural Information Processing Systems},
year = {2012},
pages = {2663-2671},
url = {https://mlanthology.org/neurips/2012/roux2012neurips-stochastic/}
}