Automatic Tuning of Stochastic Gradient Descent with Bayesian Optimisation
Abstract
Many machine learning models require a training procedure based on running stochastic gradient descent. A key element for the efficiency of those algorithms is the choice of the learning rate schedule. While finding good learning rates schedules using Bayesian optimisation has been tackled by several authors, adapting it dynamically in a data-driven way is an open question. This is of high practical importance to users that need to train a single, expensive model. To tackle this problem, we introduce an original probabilistic model for traces of optimisers, based on latent Gaussian processes and an auto-/regressive formulation, that flexibly adjusts to abrupt changes of behaviours induced by new learning rate values. As illustrated, this model is well-suited to tackle a set of problems: first, for the on-line adaptation of the learning rate for a cold-started run; then, for tuning the schedule for a set of similar tasks (in a classical BO setup), as well as warm-starting it for a new task.
Cite
Text
Picheny et al. "Automatic Tuning of Stochastic Gradient Descent with Bayesian Optimisation." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2020. doi:10.1007/978-3-030-67664-3_26Markdown
[Picheny et al. "Automatic Tuning of Stochastic Gradient Descent with Bayesian Optimisation." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2020.](https://mlanthology.org/ecmlpkdd/2020/picheny2020ecmlpkdd-automatic/) doi:10.1007/978-3-030-67664-3_26BibTeX
@inproceedings{picheny2020ecmlpkdd-automatic,
title = {{Automatic Tuning of Stochastic Gradient Descent with Bayesian Optimisation}},
author = {Picheny, Victor and Dutordoir, Vincent and Artemev, Artem and Durrande, Nicolas},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2020},
pages = {431-446},
doi = {10.1007/978-3-030-67664-3_26},
url = {https://mlanthology.org/ecmlpkdd/2020/picheny2020ecmlpkdd-automatic/}
}