Normalization in Attention Dynamics

Abstract

We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes---including **Post-LN**, **Pre-LN**, **Mix-LN**, **Peri-LN**, **nGPT**---revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying **Peri-LN** as a particularly effective choice.

PDF NeurIPS OpenReview Semantic Scholar

Cite

Text

Karagodin et al. "Normalization in Attention Dynamics." Advances in Neural Information Processing Systems, 2025.

Markdown

[Karagodin et al. "Normalization in Attention Dynamics." Advances in Neural Information Processing Systems, 2025.](https://mlanthology.org/neurips/2025/karagodin2025neurips-normalization/)

BibTeX

@inproceedings{karagodin2025neurips-normalization,
  title     = {{Normalization in Attention Dynamics}},
  author    = {Karagodin, Nikita and Ge, Shu and Polyanskiy, Yury and Rigollet, Philippe},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2025},
  url       = {https://mlanthology.org/neurips/2025/karagodin2025neurips-normalization/}
}