Identity Crisis: Memorization and Generalization Under Extreme Overparameterization
Abstract
We study the interplay between memorization and generalization of overparametrized networks in the extreme case of a single training example. The learning task is to predict an output which is as similar as possible to the input. We examine both fully-connected and convolutional networks that are initialized randomly and then trained to minimize the reconstruction error. The trained networks take one of the two forms: the constant function (``memorization'') and the identity function (``generalization''). We show that different architectures exhibit vastly different inductive bias towards memorization and generalization.
Cite
Text
Zhang et al. "Identity Crisis: Memorization and Generalization Under Extreme Overparameterization." ICML 2019 Workshops: Deep_Phenomena, 2019.Markdown
[Zhang et al. "Identity Crisis: Memorization and Generalization Under Extreme Overparameterization." ICML 2019 Workshops: Deep_Phenomena, 2019.](https://mlanthology.org/icmlw/2019/zhang2019icmlw-identity/)BibTeX
@inproceedings{zhang2019icmlw-identity,
title = {{Identity Crisis: Memorization and Generalization Under Extreme Overparameterization}},
author = {Zhang, Chiyuan and Bengio, Samy and Hardt, Moritz and Mozer, Michael C. and Singer, Yoram},
booktitle = {ICML 2019 Workshops: Deep_Phenomena},
year = {2019},
url = {https://mlanthology.org/icmlw/2019/zhang2019icmlw-identity/}
}