Meta-Optimization for Deep Learning via Nonstochastic Control
Abstract
Hyperparameter tuning in mathematical optimization is a notoriously difficult problem. Recent tools from online control give rise to a provable methodology for hyperparameter tuning in convex optimization called meta-optimization. In this work, we extend this methodology to nonconvex optimization and the training of deep neural networks. We present an algorithm for nonconvex meta-optimization that leverages the reduction from nonconvex optimization to convex optimization, and investigate its applicability for deep learning tasks on academic-scale datasets.
Cite
Text
Chen et al. "Meta-Optimization for Deep Learning via Nonstochastic Control." ICML 2024 Workshops: TF2M, 2024.Markdown
[Chen et al. "Meta-Optimization for Deep Learning via Nonstochastic Control." ICML 2024 Workshops: TF2M, 2024.](https://mlanthology.org/icmlw/2024/chen2024icmlw-metaoptimization/)BibTeX
@inproceedings{chen2024icmlw-metaoptimization,
title = {{Meta-Optimization for Deep Learning via Nonstochastic Control}},
author = {Chen, Xinyi and Dogariu, Evan and Lu, Zhou and Hazan, Elad},
booktitle = {ICML 2024 Workshops: TF2M},
year = {2024},
url = {https://mlanthology.org/icmlw/2024/chen2024icmlw-metaoptimization/}
}