CDF Normalization for Controlling the Distribution of Hidden Layer Activations

Abstract

Batch Normalizaiton (BN) is a normalization method for deep neural networks that has been shown to accelerate training. While the effectiveness of BN is undisputed, the explanation of its effectiveness is still being studied. The original BN paper attributes the success of BN to reducing internal covariate shift, so we take this a step further and explicitly enforce a Gaussian distribution on hidden layer activations. This approach proves to be ineffective, demonstrating further that reducing internal covariate shift is not important for successful layer normalization.

PDF NeurIPSW OpenReview Semantic Scholar

Cite

Text

Van Ness and Udell. "CDF Normalization for Controlling the Distribution of Hidden Layer Activations." NeurIPS 2021 Workshops: ICBINB, 2021.

Markdown

[Van Ness and Udell. "CDF Normalization for Controlling the Distribution of Hidden Layer Activations." NeurIPS 2021 Workshops: ICBINB, 2021.](https://mlanthology.org/neuripsw/2021/ness2021neuripsw-cdf/)

BibTeX

@inproceedings{ness2021neuripsw-cdf,
  title     = {{CDF Normalization for Controlling the Distribution of Hidden Layer Activations}},
  author    = {Van Ness, Mike and Udell, Madeleine},
  booktitle = {NeurIPS 2021 Workshops: ICBINB},
  year      = {2021},
  url       = {https://mlanthology.org/neuripsw/2021/ness2021neuripsw-cdf/}
}