On Hyperparameter Optimization in Learning Systems
Abstract
We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm. These procedures mirror two ways of computing gradients for recurrent neural networks and have different trade-offs in terms of running time and space requirements. The reverse-mode procedure extends previous work by (Maclaurin et al. 2015) and offers the opportunity to insert constraints on the hyperparameters in a natural way. The forward-mode procedure is suitable for stochastic hyperparameter updates, which may significantly speedup the overall hyperparameter optimization procedure.
Cite
Text
Franceschi et al. "On Hyperparameter Optimization in Learning Systems." International Conference on Learning Representations, 2017.Markdown
[Franceschi et al. "On Hyperparameter Optimization in Learning Systems." International Conference on Learning Representations, 2017.](https://mlanthology.org/iclr/2017/franceschi2017iclr-hyperparameter/)BibTeX
@inproceedings{franceschi2017iclr-hyperparameter,
title = {{On Hyperparameter Optimization in Learning Systems}},
author = {Franceschi, Luca and Donini, Michele and Frasconi, Paolo and Pontil, Massimiliano},
booktitle = {International Conference on Learning Representations},
year = {2017},
url = {https://mlanthology.org/iclr/2017/franceschi2017iclr-hyperparameter/}
}