Recent Advances on Neural Network Pruning at Initialization
Abstract
Neural network pruning typically removes connections or neurons from a pretrained converged model; while a new pruning paradigm, pruning at initialization (PaI), attempts to prune a randomly initialized network. This paper offers the first survey concentrated on this emerging pruning fashion. We first introduce a generic formulation of neural network pruning, followed by the major classic pruning topics. Then, as the main body of this paper, a thorough and structured literature review of PaI methods is presented, consisting of two major tracks (sparse training and sparse selection). Finally, we summarize the surge of PaI compared to PaT and discuss the open problems. Apart from the dedicated literature review, this paper also offers a code base for easy sanity-checking and benchmarking of different PaI methods.
Cite
Text
Wang et al. "Recent Advances on Neural Network Pruning at Initialization." International Joint Conference on Artificial Intelligence, 2022. doi:10.24963/IJCAI.2022/786Markdown
[Wang et al. "Recent Advances on Neural Network Pruning at Initialization." International Joint Conference on Artificial Intelligence, 2022.](https://mlanthology.org/ijcai/2022/wang2022ijcai-recent/) doi:10.24963/IJCAI.2022/786BibTeX
@inproceedings{wang2022ijcai-recent,
title = {{Recent Advances on Neural Network Pruning at Initialization}},
author = {Wang, Huan and Qin, Can and Bai, Yue and Zhang, Yulun and Fu, Yun},
booktitle = {International Joint Conference on Artificial Intelligence},
year = {2022},
pages = {5638-5645},
doi = {10.24963/IJCAI.2022/786},
url = {https://mlanthology.org/ijcai/2022/wang2022ijcai-recent/}
}