Better Parameter-Free Stochastic Optimization with ODE Updates for Coin-Betting
Abstract
Parameter-free stochastic gradient descent (PFSGD) algorithms do not require setting learning rates while achieving optimal theoretical performance. In practical applications, however, there remains an empirical gap between tuned stochastic gradient descent (SGD) and PFSGD. In this paper, we close the empirical gap with a new parameter-free algorithm based on continuous-time Coin-Betting on truncated models. The new update is derived through the solution of an Ordinary Differential Equation (ODE) and solved in a closed form. We show empirically that this new parameter-free algorithm outperforms algorithms with the ``best default'' learning rates and almost matches the performance of finely tuned baselines without anything to tune.
Cite
Text
Chen et al. "Better Parameter-Free Stochastic Optimization with ODE Updates for Coin-Betting." AAAI Conference on Artificial Intelligence, 2022. doi:10.1609/AAAI.V36I6.20573Markdown
[Chen et al. "Better Parameter-Free Stochastic Optimization with ODE Updates for Coin-Betting." AAAI Conference on Artificial Intelligence, 2022.](https://mlanthology.org/aaai/2022/chen2022aaai-better/) doi:10.1609/AAAI.V36I6.20573BibTeX
@inproceedings{chen2022aaai-better,
title = {{Better Parameter-Free Stochastic Optimization with ODE Updates for Coin-Betting}},
author = {Chen, Keyi and Langford, John and Orabona, Francesco},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2022},
pages = {6239-6247},
doi = {10.1609/AAAI.V36I6.20573},
url = {https://mlanthology.org/aaai/2022/chen2022aaai-better/}
}