A Theoretical Study of the $(L_0,L_1)$-Smoothness Condition in Deep Learning
Abstract
We study the $(L_0,L_1)$-smoothness condition introduced by Zhang-He-Sra-Jadbabai in 2020 in the setting of loss functions arising in deep learning. Theoretical work on $(L_0,L_1)$-smoothnes has focused on convergence guarantees for functions which satisfy this condition. In this paper we provide theoretical analysis of the condition in the setting of feedforward neural networks of depth at least 2, with either $L2$ or cross-entropy loss, and find the $(L_0,L_1)$-smoothness condition is not satisfied.
Cite
Text
Cooper. "A Theoretical Study of the $(L_0,L_1)$-Smoothness Condition in Deep Learning." NeurIPS 2024 Workshops: OPT, 2024.Markdown
[Cooper. "A Theoretical Study of the $(L_0,L_1)$-Smoothness Condition in Deep Learning." NeurIPS 2024 Workshops: OPT, 2024.](https://mlanthology.org/neuripsw/2024/cooper2024neuripsw-theoretical/)BibTeX
@inproceedings{cooper2024neuripsw-theoretical,
title = {{A Theoretical Study of the $(L_0,L_1)$-Smoothness Condition in Deep Learning}},
author = {Cooper, Y},
booktitle = {NeurIPS 2024 Workshops: OPT},
year = {2024},
url = {https://mlanthology.org/neuripsw/2024/cooper2024neuripsw-theoretical/}
}