Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks
Abstract
In this paper, we apply harmonic distortion analysis to understand the effect of nonlinearities in the spectral domain. Each nonlinear layer creates higher-frequency harmonics, which we call "blueshift", whose magnitude increases with network depth, thereby increasing the “roughness” of the output landscape. Unlike differential models (such as vanishing gradients, sharpness), this provides a more global view of how network architectures behave across larger areas of their parameter domain. For example, the model predicts that residual connections are able to counter the effect by dampening corresponding higher frequency modes. We empirically verify the connection between blueshift and architectural choices, and provide evidence for a connection with trainability.
Cite
Text
Mehmeti-Göpel et al. "Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks." International Conference on Learning Representations, 2021.Markdown
[Mehmeti-Göpel et al. "Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks." International Conference on Learning Representations, 2021.](https://mlanthology.org/iclr/2021/mehmetigopel2021iclr-ringing/)BibTeX
@inproceedings{mehmetigopel2021iclr-ringing,
title = {{Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks}},
author = {Mehmeti-Göpel, Christian H.X. Ali and Hartmann, David and Wand, Michael},
booktitle = {International Conference on Learning Representations},
year = {2021},
url = {https://mlanthology.org/iclr/2021/mehmetigopel2021iclr-ringing/}
}