More General and Effective Model Compression via an Additive Combination of Compressions
Abstract
Model compression is generally performed by using quantization, low-rank approximation or pruning, for which various algorithms have been researched in recent years. One fundamental question is: what types of compression work better for a given model? Or even better: can we improve by combining compressions in a suitable way? We formulate this generally as a problem of optimizing the loss but where the weights are constrained to equal an additive combination of separately compressed parts; and we give an algorithm to learn the corresponding parts’ parameters. Experimentally with deep neural nets, we observe that 1) we can find significantly better models in the error-compression space, indicating that different compression types have complementary benefits, and 2) the best type of combination depends exquisitely on the type of neural net. For example, we can compress ResNets and AlexNet using only 1 bit per weight without error degradation at the cost of adding a few floating point weights. However, VGG nets can be better compressed by combining low-rank with a few floating point weights.
Cite
Text
Idelbayev and Carreira-Perpiñán. "More General and Effective Model Compression via an Additive Combination of Compressions." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2021. doi:10.1007/978-3-030-86523-8_15Markdown
[Idelbayev and Carreira-Perpiñán. "More General and Effective Model Compression via an Additive Combination of Compressions." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2021.](https://mlanthology.org/ecmlpkdd/2021/idelbayev2021ecmlpkdd-more/) doi:10.1007/978-3-030-86523-8_15BibTeX
@inproceedings{idelbayev2021ecmlpkdd-more,
title = {{More General and Effective Model Compression via an Additive Combination of Compressions}},
author = {Idelbayev, Yerlan and Carreira-Perpiñán, Miguel Á.},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2021},
pages = {233-248},
doi = {10.1007/978-3-030-86523-8_15},
url = {https://mlanthology.org/ecmlpkdd/2021/idelbayev2021ecmlpkdd-more/}
}