On the Implicit Bias of Dropout
Abstract
Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For single hidden-layer linear neural networks, we show that dropout tends to make the norm of incoming/outgoing weight vectors of all the hidden nodes equal. In addition, we provide a complete characterization of the optimization landscape induced by dropout.
Cite
Text
Mianjy et al. "On the Implicit Bias of Dropout." International Conference on Machine Learning, 2018.Markdown
[Mianjy et al. "On the Implicit Bias of Dropout." International Conference on Machine Learning, 2018.](https://mlanthology.org/icml/2018/mianjy2018icml-implicit/)BibTeX
@inproceedings{mianjy2018icml-implicit,
title = {{On the Implicit Bias of Dropout}},
author = {Mianjy, Poorya and Arora, Raman and Vidal, Rene},
booktitle = {International Conference on Machine Learning},
year = {2018},
pages = {3540-3548},
volume = {80},
url = {https://mlanthology.org/icml/2018/mianjy2018icml-implicit/}
}