Practical Structured Riemannian Optimization with Momentum by Using Generalized Normal Coordinates
Abstract
Adding momentum into Riemannian optimization is computationally challenging due to the intractable ODEs needed to define the exponential and parallel transport maps. We address these issues for Gaussian Fisher-Rao manifolds by proposing new local coordinates to exploit sparse structures and efficiently approximate the ODEs, which results in a numerically stable update scheme. Our approach extends the structured natural-gradient descent method of Lin et al. (2021a) by incorporating momentum into it and scaling the method for large-scale applications arising in numerical optimization and deep learning.
Cite
Text
Lin et al. "Practical Structured Riemannian Optimization with Momentum by Using Generalized Normal Coordinates." NeurIPS 2022 Workshops: NeurReps, 2022.Markdown
[Lin et al. "Practical Structured Riemannian Optimization with Momentum by Using Generalized Normal Coordinates." NeurIPS 2022 Workshops: NeurReps, 2022.](https://mlanthology.org/neuripsw/2022/lin2022neuripsw-practical/)BibTeX
@inproceedings{lin2022neuripsw-practical,
title = {{Practical Structured Riemannian Optimization with Momentum by Using Generalized Normal Coordinates}},
author = {Lin, Wu and Duruisseaux, Valentin and Leok, Melvin and Nielsen, Frank and Khan, Mohammad Emtiyaz and Schmidt, Mark},
booktitle = {NeurIPS 2022 Workshops: NeurReps},
year = {2022},
url = {https://mlanthology.org/neuripsw/2022/lin2022neuripsw-practical/}
}