A Class of Short-Term Recurrence Anderson Mixing Methods and Their Applications
Abstract
Anderson mixing (AM) is a powerful acceleration method for fixed-point iterations, but its computation requires storing many historical iterations. The extra memory footprint can be prohibitive when solving high-dimensional problems in a resource-limited machine. To reduce the memory overhead, we propose a novel class of short-term recurrence AM methods (ST-AM). The ST-AM methods only store two previous iterations with cheap corrections. We prove that the basic version of ST-AM is equivalent to the full-memory AM in strongly convex quadratic optimization, and with minor changes it has local linear convergence for solving general nonlinear fixed-point problems. We further analyze the convergence properties of the regularized ST-AM for nonconvex (stochastic) optimization. Finally, we apply ST-AM to several applications including solving root-finding problems and training neural networks. Experimental results show that ST-AM is competitive with the long-memory AM and outperforms many existing optimizers.
Cite
Text
Wei et al. "A Class of Short-Term Recurrence Anderson Mixing Methods and Their Applications." International Conference on Learning Representations, 2022.Markdown
[Wei et al. "A Class of Short-Term Recurrence Anderson Mixing Methods and Their Applications." International Conference on Learning Representations, 2022.](https://mlanthology.org/iclr/2022/wei2022iclr-class/)BibTeX
@inproceedings{wei2022iclr-class,
title = {{A Class of Short-Term Recurrence Anderson Mixing Methods and Their Applications}},
author = {Wei, Fuchao and Bao, Chenglong and Liu, Yang},
booktitle = {International Conference on Learning Representations},
year = {2022},
url = {https://mlanthology.org/iclr/2022/wei2022iclr-class/}
}