Accelerating AI Performance Using Anderson Extrapolation on GPUs
Abstract
We present a novel approach for accelerating AI performance by leveraging Anderson extrapolation, a vector-to-vector mapping technique based on a window of historical iterations. By identifying the crossover point (Fig. 1) where a mixing penalty is incurred, the method focuses on reducing iterations to convergence, with fewer more compute-intensive but generally cacheable iterations, balancing speed and memory usage with accuracy and algorithmic stability, respectively. We demonstrate significant improvements in both training and inference, motivated by scalability and efficiency extensions to the realm of high-performance computing (HPC).
Cite
Text
Al Dajani and Keyes. "Accelerating AI Performance Using Anderson Extrapolation on GPUs." NeurIPS 2024 Workshops: MLNCP, 2024.Markdown
[Al Dajani and Keyes. "Accelerating AI Performance Using Anderson Extrapolation on GPUs." NeurIPS 2024 Workshops: MLNCP, 2024.](https://mlanthology.org/neuripsw/2024/dajani2024neuripsw-accelerating/)BibTeX
@inproceedings{dajani2024neuripsw-accelerating,
title = {{Accelerating AI Performance Using Anderson Extrapolation on GPUs}},
author = {Al Dajani, Saleem Abdul Fattah Ahmed and Keyes, David},
booktitle = {NeurIPS 2024 Workshops: MLNCP},
year = {2024},
url = {https://mlanthology.org/neuripsw/2024/dajani2024neuripsw-accelerating/}
}