Progressive Blockwise Knowledge Distillation for Neural Network Acceleration
Abstract
As an important and challenging problem in machine learning and computer vision, neural network acceleration essentially aims to enhance the computational efficiency without sacrificing the model accuracy too much. In this paper, we propose a progressive blockwise learning scheme for teacher-student model distillation at the subnetwork block level. The proposed scheme is able to distill the knowledge of the entire teacher network by locally extracting the knowledge of each block in terms of progressive blockwise function approximation. Furthermore, we propose a structure design criterion for the student subnetwork block, which is able to effectively preserve the original receptive field from the teacher network. Experimental results demonstrate the effectiveness of the proposed scheme against the state-of-the-art approaches.
Cite
Text
Wang et al. "Progressive Blockwise Knowledge Distillation for Neural Network Acceleration." International Joint Conference on Artificial Intelligence, 2018. doi:10.24963/IJCAI.2018/384Markdown
[Wang et al. "Progressive Blockwise Knowledge Distillation for Neural Network Acceleration." International Joint Conference on Artificial Intelligence, 2018.](https://mlanthology.org/ijcai/2018/wang2018ijcai-progressive/) doi:10.24963/IJCAI.2018/384BibTeX
@inproceedings{wang2018ijcai-progressive,
title = {{Progressive Blockwise Knowledge Distillation for Neural Network Acceleration}},
author = {Wang, Hui and Zhao, Hanbin and Li, Xi and Tan, Xu},
booktitle = {International Joint Conference on Artificial Intelligence},
year = {2018},
pages = {2769-2775},
doi = {10.24963/IJCAI.2018/384},
url = {https://mlanthology.org/ijcai/2018/wang2018ijcai-progressive/}
}