Overparameterization Implicitly Regularizes Input-Space Smoothness

Abstract

Existing bounds on the generalization error of deep networks assume some form of smooth or bounded dependence on the input variable and intermediate activations, falling short of investigating the mechanisms controlling such factors in practice. In this work, we present an empirical study of the Lipschitz constant of networks trained in practice, as the number of model parameters and training epochs vary. We present non-monotonic trends for the Lipschitz constant, strongly correlating with double descent for the test error. Our findings highlight a theoretical shortcoming in modeling input-space smoothness via uniform bounds.

PDF NeurIPSW OpenReview Semantic Scholar

Cite

Text

Gamba et al. "Overparameterization Implicitly Regularizes Input-Space Smoothness." NeurIPS 2022 Workshops: INTERPOLATE, 2022.

Markdown

[Gamba et al. "Overparameterization Implicitly Regularizes Input-Space Smoothness." NeurIPS 2022 Workshops: INTERPOLATE, 2022.](https://mlanthology.org/neuripsw/2022/gamba2022neuripsw-overparameterization/)

BibTeX

@inproceedings{gamba2022neuripsw-overparameterization,
  title     = {{Overparameterization Implicitly Regularizes Input-Space Smoothness}},
  author    = {Gamba, Matteo and Azizpour, Hossein and Björkman, Mårten},
  booktitle = {NeurIPS 2022 Workshops: INTERPOLATE},
  year      = {2022},
  url       = {https://mlanthology.org/neuripsw/2022/gamba2022neuripsw-overparameterization/}
}