Normalization Matters for Optimization Performance on Graph Neural Networks

Abstract

We show that feature normalization has a drastic impact on the performance of optimization algorithms in the context of graph neural networks. The standard normalization scheme used throughout the graph neural network literature is not motivated from an optimization perspective, and leads (S)GD to frequently fail. Adam does not fail, but is also negatively impacted by standard normalization methods. We show across multiple datasets and models that better motivated feature normalization closes the gap between Adam and (S)GD, and speeds up optimization for both.

PDF NeurIPSW OpenReview Semantic Scholar

Cite

Text

Milligan et al. "Normalization Matters for Optimization Performance on Graph Neural Networks." NeurIPS 2024 Workshops: OPT, 2024.

Markdown

[Milligan et al. "Normalization Matters for Optimization Performance on Graph Neural Networks." NeurIPS 2024 Workshops: OPT, 2024.](https://mlanthology.org/neuripsw/2024/milligan2024neuripsw-normalization/)

BibTeX

@inproceedings{milligan2024neuripsw-normalization,
  title     = {{Normalization Matters for Optimization Performance on Graph Neural Networks}},
  author    = {Milligan, Alan and Kunstner, Frederik and Shirzad, Hamed and Schmidt, Mark and Sutherland, Danica J.},
  booktitle = {NeurIPS 2024 Workshops: OPT},
  year      = {2024},
  url       = {https://mlanthology.org/neuripsw/2024/milligan2024neuripsw-normalization/}
}