Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent
Abstract
We present a direct (primal only) derivation of Mirror Descent as a “partial” discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential function. We contrast this discretization to Natural Gradient Descent, which is obtained by a “full” forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to any Riemannian geometry in $\mathbb{R}^d$, even when the metric tensor is not a Hessian, and thus there is no “dual.”
Cite
Text
Gunasekar et al. " Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent ." Artificial Intelligence and Statistics, 2021.Markdown
[Gunasekar et al. " Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent ." Artificial Intelligence and Statistics, 2021.](https://mlanthology.org/aistats/2021/gunasekar2021aistats-mirrorless/)BibTeX
@inproceedings{gunasekar2021aistats-mirrorless,
title = {{ Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent }},
author = {Gunasekar, Suriya and Woodworth, Blake and Srebro, Nathan},
booktitle = {Artificial Intelligence and Statistics},
year = {2021},
pages = {2305-2313},
volume = {130},
url = {https://mlanthology.org/aistats/2021/gunasekar2021aistats-mirrorless/}
}