Hot Swapping for Online Adaptation of Optimization Hyperparameters
Abstract
We describe a general framework for online adaptation of optimization hyperparameters by `hot swapping' their values during learning. We investigate this approach in the context of adaptive learning rate selection using an explore-exploit strategy from the multi-armed bandit literature. Experiments on a benchmark neural network show that the hot swapping approach leads to consistently better solutions compared to well-known alternatives such as AdaDelta and stochastic gradient with exhaustive hyperparameter search.
Cite
Text
Bache et al. "Hot Swapping for Online Adaptation of Optimization Hyperparameters." International Conference on Learning Representations, 2015.Markdown
[Bache et al. "Hot Swapping for Online Adaptation of Optimization Hyperparameters." International Conference on Learning Representations, 2015.](https://mlanthology.org/iclr/2015/bache2015iclr-hot/)BibTeX
@inproceedings{bache2015iclr-hot,
title = {{Hot Swapping for Online Adaptation of Optimization Hyperparameters}},
author = {Bache, Kevin and DeCoste, Dennis and Smyth, Padhraic},
booktitle = {International Conference on Learning Representations},
year = {2015},
url = {https://mlanthology.org/iclr/2015/bache2015iclr-hot/}
}