How Learning Rates Shape Neural Network Focus: Insights from Example Ranking

Abstract

The learning rate is a key hyperparameter that affects both the speed of training and the generalization performance of neural networks. Through a new {\it loss-based example ranking} analysis, we show that networks trained with different learning rates focus their capacity on different parts of the data distribution, leading to solutions with different generalization properties. These findings, which hold across architectures and datasets, provide new insights into how learning rates affect model performance and example-level dynamics in neural networks.

PDF NeurIPSW OpenReview Semantic Scholar

Cite

Text

Lobacheva et al. "How Learning Rates Shape Neural Network Focus: Insights from Example Ranking." NeurIPS 2024 Workshops: SciForDL, 2024.

Markdown

[Lobacheva et al. "How Learning Rates Shape Neural Network Focus: Insights from Example Ranking." NeurIPS 2024 Workshops: SciForDL, 2024.](https://mlanthology.org/neuripsw/2024/lobacheva2024neuripsw-learning/)

BibTeX

@inproceedings{lobacheva2024neuripsw-learning,
  title     = {{How Learning Rates Shape Neural Network Focus: Insights from Example Ranking}},
  author    = {Lobacheva, Ekaterina and Jordan, Keller and Baratin, Aristide and Le Roux, Nicolas},
  booktitle = {NeurIPS 2024 Workshops: SciForDL},
  year      = {2024},
  url       = {https://mlanthology.org/neuripsw/2024/lobacheva2024neuripsw-learning/}
}