Amortized Nesterov’s Momentum: A Robust Momentum and Its Application to Deep Learning
Abstract
This work proposes a novel momentum technique, the Amortized Nesterov’s Momentum, for stochastic convex optimization. The proposed method can be regarded as a smooth transition between Nesterov’s method and mirror descent. By tuning only a single parameter, users can trade Nesterov’s acceleration for robustness, that is, the variance control of the stochastic noise. Motivated by the recent success of using momentum in deep learning, we conducted extensive experiments to evaluate this new momentum in deep learning tasks. The results suggest that it can serve as a favorable alternative for Nesterov’s momentum.
Cite
Text
Zhou et al. "Amortized Nesterov’s Momentum: A Robust Momentum and Its Application to Deep Learning." Uncertainty in Artificial Intelligence, 2020.Markdown
[Zhou et al. "Amortized Nesterov’s Momentum: A Robust Momentum and Its Application to Deep Learning." Uncertainty in Artificial Intelligence, 2020.](https://mlanthology.org/uai/2020/zhou2020uai-amortized/)BibTeX
@inproceedings{zhou2020uai-amortized,
title = {{Amortized Nesterov’s Momentum: A Robust Momentum and Its Application to Deep Learning}},
author = {Zhou, Kaiwen and Jin, Yanghua and Ding, Qinghua and Cheng, James},
booktitle = {Uncertainty in Artificial Intelligence},
year = {2020},
pages = {211-220},
volume = {124},
url = {https://mlanthology.org/uai/2020/zhou2020uai-amortized/}
}