Global Optimality of Elman-Type RNNs in the Mean-Field Regime
Abstract
We analyze Elman-type recurrent neural networks (RNNs) and their training in the mean-field regime. Specifically, we show convergence of gradient descent training dynamics of the RNN to the corresponding mean-field formulation in the large width limit. We also show that the fixed points of the limiting infinite-width dynamics are globally optimal, under some assumptions on the initialization of the weights. Our results establish optimality for feature-learning with wide RNNs in the mean-field regime.
Cite
Text
Agazzi et al. "Global Optimality of Elman-Type RNNs in the Mean-Field Regime." International Conference on Machine Learning, 2023.Markdown
[Agazzi et al. "Global Optimality of Elman-Type RNNs in the Mean-Field Regime." International Conference on Machine Learning, 2023.](https://mlanthology.org/icml/2023/agazzi2023icml-global/)BibTeX
@inproceedings{agazzi2023icml-global,
title = {{Global Optimality of Elman-Type RNNs in the Mean-Field Regime}},
author = {Agazzi, Andrea and Lu, Jianfeng and Mukherjee, Sayan},
booktitle = {International Conference on Machine Learning},
year = {2023},
pages = {196-227},
volume = {202},
url = {https://mlanthology.org/icml/2023/agazzi2023icml-global/}
}