Dropout Regularization Versus L2-Penalization in the Linear Model
Abstract
We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and $\ell_2$-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study a simplified variant of dropout which does not have a regularizing effect and converges to the least squares estimator.
Cite
Text
Clara et al. "Dropout Regularization Versus L2-Penalization in the Linear Model." Journal of Machine Learning Research, 2024.Markdown
[Clara et al. "Dropout Regularization Versus L2-Penalization in the Linear Model." Journal of Machine Learning Research, 2024.](https://mlanthology.org/jmlr/2024/clara2024jmlr-dropout/)BibTeX
@article{clara2024jmlr-dropout,
title = {{Dropout Regularization Versus L2-Penalization in the Linear Model}},
author = {Clara, Gabriel and Langer, Sophie and Schmidt-Hieber, Johannes},
journal = {Journal of Machine Learning Research},
year = {2024},
pages = {1-48},
volume = {25},
url = {https://mlanthology.org/jmlr/2024/clara2024jmlr-dropout/}
}