Neural Networks Are Convex Regularizers: Exact Polynomial-Time Convex Optimization Formulations for Two-Layer Networks
Abstract
We develop exact representations of training two-layer neural networks with rectified linear units (ReLUs) in terms of a single convex program with number of variables polynomial in the number of training samples and the number of hidden neurons. Our theory utilizes semi-infinite duality and minimum norm regularization. We show that ReLU networks trained with standard weight decay are equivalent to block $\ell_1$ penalized convex models. Moreover, we show that certain standard convolutional linear networks are equivalent semi-definite programs which can be simplified to $\ell_1$ regularized linear models in a polynomial sized discrete Fourier feature space
Cite
Text
Pilanci and Ergen. "Neural Networks Are Convex Regularizers: Exact Polynomial-Time Convex Optimization Formulations for Two-Layer Networks." International Conference on Machine Learning, 2020.Markdown
[Pilanci and Ergen. "Neural Networks Are Convex Regularizers: Exact Polynomial-Time Convex Optimization Formulations for Two-Layer Networks." International Conference on Machine Learning, 2020.](https://mlanthology.org/icml/2020/pilanci2020icml-neural/)BibTeX
@inproceedings{pilanci2020icml-neural,
title = {{Neural Networks Are Convex Regularizers: Exact Polynomial-Time Convex Optimization Formulations for Two-Layer Networks}},
author = {Pilanci, Mert and Ergen, Tolga},
booktitle = {International Conference on Machine Learning},
year = {2020},
pages = {7695-7705},
volume = {119},
url = {https://mlanthology.org/icml/2020/pilanci2020icml-neural/}
}