Equi-Normalization of Neural Networks
Abstract
Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and out- put weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the l2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch- and group- normalization on CIFAR-10 and ImageNet with a ResNet-18.
Cite
Text
Stock et al. "Equi-Normalization of Neural Networks." International Conference on Learning Representations, 2019.Markdown
[Stock et al. "Equi-Normalization of Neural Networks." International Conference on Learning Representations, 2019.](https://mlanthology.org/iclr/2019/stock2019iclr-equinormalization/)BibTeX
@inproceedings{stock2019iclr-equinormalization,
title = {{Equi-Normalization of Neural Networks}},
author = {Stock, Pierre and Graham, Benjamin and Gribonval, Rémi and Jégou, Hervé},
booktitle = {International Conference on Learning Representations},
year = {2019},
url = {https://mlanthology.org/iclr/2019/stock2019iclr-equinormalization/}
}