Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
Abstract
We provide a detailed asymptotic study of gradient flow trajectories and their implicit optimization bias when minimizing the exponential loss over "diagonal linear networks". This is the simplest model displaying a transition between "kernel" and non-kernel ("rich" or "active") regimes. We show how the transition is controlled by the relationship between the initialization scale and how accurately we minimize the training loss. Our results indicate that some limit behavior of gradient descent only kick in at ridiculous training accuracies (well beyond 10^-100). Moreover, the implicit bias at reasonable initialization scales and training accuracies is more complex and not captured by these limits.
Cite
Text
Moroshko et al. "Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy." Neural Information Processing Systems, 2020.Markdown
[Moroshko et al. "Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy." Neural Information Processing Systems, 2020.](https://mlanthology.org/neurips/2020/moroshko2020neurips-implicit/)BibTeX
@inproceedings{moroshko2020neurips-implicit,
title = {{Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy}},
author = {Moroshko, Edward and Woodworth, Blake E and Gunasekar, Suriya and Lee, Jason and Srebro, Nati and Soudry, Daniel},
booktitle = {Neural Information Processing Systems},
year = {2020},
url = {https://mlanthology.org/neurips/2020/moroshko2020neurips-implicit/}
}