Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer
Abstract
Neural net compression can be achieved by approximating each layer's weight matrix by a low-rank matrix. The real difficulty in doing this is not in training the resulting neural net (made up of one low-rank matrix per layer), but in determining what the optimal rank of each layer is--effectively, an architecture search problem with one hyperparameter per layer. We show that, with a suitable formulation, this problem is amenable to a mixed discrete-continuous optimization jointly over the ranks and over the matrix elements, and give a corresponding algorithm. We show that this indeed can select ranks much better than existing approaches, making low-rank compression much more attractive than previously thought. For example, we can make a VGG network faster than a ResNet and with nearly the same classification error.
Cite
Text
Idelbayev and Carreira-Perpinan. "Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. doi:10.1109/CVPR42600.2020.00807Markdown
[Idelbayev and Carreira-Perpinan. "Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.](https://mlanthology.org/cvpr/2020/idelbayev2020cvpr-lowrank/) doi:10.1109/CVPR42600.2020.00807BibTeX
@inproceedings{idelbayev2020cvpr-lowrank,
title = {{Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer}},
author = {Idelbayev, Yerlan and Carreira-Perpinan, Miguel A.},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2020},
doi = {10.1109/CVPR42600.2020.00807},
url = {https://mlanthology.org/cvpr/2020/idelbayev2020cvpr-lowrank/}
}