Explaining Knowledge Distillation by Quantifying the Knowledge
Abstract
This paper presents a method to interpret the success of knowledge distillation by quantifying and analyzing task-relevant and task-irrelevant visual concepts that are encoded in intermediate layers of a deep neural network (DNN). More specifically, three hypotheses are proposed as follows. 1. Knowledge distillation makes the DNN learn more visual concepts than learning from raw data. 2. Knowledge distillation ensures that the DNN is prone to learning various visual concepts simultaneously. Whereas, in the scenario of learning from raw data, the DNN learns visual concepts sequentially. 3. Knowledge distillation yields more stable optimization directions than learning from raw data. Accordingly, we design three types of mathematical metrics to evaluate feature representations of the DNN. In experiments, we diagnosed various DNNs, and above hypotheses were verified.
Cite
Text
Cheng et al. "Explaining Knowledge Distillation by Quantifying the Knowledge." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. doi:10.1109/CVPR42600.2020.01294Markdown
[Cheng et al. "Explaining Knowledge Distillation by Quantifying the Knowledge." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.](https://mlanthology.org/cvpr/2020/cheng2020cvpr-explaining/) doi:10.1109/CVPR42600.2020.01294BibTeX
@inproceedings{cheng2020cvpr-explaining,
title = {{Explaining Knowledge Distillation by Quantifying the Knowledge}},
author = {Cheng, Xu and Rao, Zhefan and Chen, Yilan and Zhang, Quanshi},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2020},
doi = {10.1109/CVPR42600.2020.01294},
url = {https://mlanthology.org/cvpr/2020/cheng2020cvpr-explaining/}
}