What if Neural Networks Had SVDs?
Abstract
Various Neural Networks employ time-consuming matrix operations like matrix inversion. Many such matrix operations are faster to compute given the Singular Value Decomposition (SVD). Techniques from (Zhang et al., 2018; Mhammedi et al., 2017) allow using the SVD in Neural Networks without computing it. In theory, the techniques can speed up matrix operations, however, in practice, they are not fast enough. We present an algorithm that is fast enough to speed up several matrix operations. The algorithm increases the degree of parallelism of an underlying matrix multiplication H*X where H is an orthogonal matrix represented by a product of Householder matrices.
Cite
Text
Mathiasen et al. "What if Neural Networks Had SVDs?." Neural Information Processing Systems, 2020.Markdown
[Mathiasen et al. "What if Neural Networks Had SVDs?." Neural Information Processing Systems, 2020.](https://mlanthology.org/neurips/2020/mathiasen2020neurips-neural/)BibTeX
@inproceedings{mathiasen2020neurips-neural,
title = {{What if Neural Networks Had SVDs?}},
author = {Mathiasen, Alexander and Hvilshøj, Frederik and Jørgensen, Jakob Rødsgaard and Nasery, Anshul and Mottin, Davide},
booktitle = {Neural Information Processing Systems},
year = {2020},
url = {https://mlanthology.org/neurips/2020/mathiasen2020neurips-neural/}
}