Stochastic Training Is Not Necessary for Generalization
Abstract
It is widely believed that the implicit regularization of SGD is fundamental to the impressive generalization behavior we observe in neural networks. In this work, we demonstrate that non-stochastic full-batch training can achieve comparably strong performance to SGD on CIFAR-10 using modern architectures. To this end, we show that the implicit regularization of SGD can be completely replaced with explicit regularization. Our observations indicate that the perceived difficulty of full-batch training may be the result of its optimization properties and the disproportionate time and effort spent by the ML community tuning optimizers and hyperparameters for small-batch training.
Cite
Text
Geiping et al. "Stochastic Training Is Not Necessary for Generalization." International Conference on Learning Representations, 2022.Markdown
[Geiping et al. "Stochastic Training Is Not Necessary for Generalization." International Conference on Learning Representations, 2022.](https://mlanthology.org/iclr/2022/geiping2022iclr-stochastic/)BibTeX
@inproceedings{geiping2022iclr-stochastic,
title = {{Stochastic Training Is Not Necessary for Generalization}},
author = {Geiping, Jonas and Goldblum, Micah and Pope, Phil and Moeller, Michael and Goldstein, Tom},
booktitle = {International Conference on Learning Representations},
year = {2022},
url = {https://mlanthology.org/iclr/2022/geiping2022iclr-stochastic/}
}