Predicting Generalization with Degrees of Freedom in Neural Networks

Abstract

Model complexity is fundamentally tied to predictive power in the sciences as well as in applications. However, there is a divergence between naive measures of complexity such as parameter count and the generalization performance of over-parameterized machine learning models. Prior empirical approaches to capturing intrinsic complexity in a more sophisticated manner than parameter count are computationally intractable, do not capture the implicitly regularizing effects of the entire machine-learning pipeline, or do not provide a quantitative fit to the double descent behavior of overparameterized models. In this work, we introduce an empirical complexity measure inspired by the classical notion of generalized degrees of freedom in statistics. This measure can be approximated efficiently and is a function of the entire machine learning training pipeline. We demonstrate that this measure correlates with generalization performance in the double-descent regime.