Finding the Optimal Network Depth in Classification Tasks
Abstract
We develop a fast end-to-end method for training lightweight neural networks using multiple classifier heads. By allowing the model to determine the importance of each head and rewarding the choice of a single shallow classifier, we are able to detect and remove unneeded components of the network. This operation, which can be seen as finding the optimal depth of the model, significantly reduces the number of parameters and accelerates inference across different hardware processing units, which is not the case for many standard pruning methods. We show the performance of our method on multiple network architectures and datasets, analyze its optimization properties, and conduct ablation studies.
Cite
Text
Wójcik et al. "Finding the Optimal Network Depth in Classification Tasks." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2020. doi:10.1007/978-3-030-67664-3_16Markdown
[Wójcik et al. "Finding the Optimal Network Depth in Classification Tasks." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2020.](https://mlanthology.org/ecmlpkdd/2020/wojcik2020ecmlpkdd-finding/) doi:10.1007/978-3-030-67664-3_16BibTeX
@inproceedings{wojcik2020ecmlpkdd-finding,
title = {{Finding the Optimal Network Depth in Classification Tasks}},
author = {Wójcik, Bartosz and Wolczyk, Maciej and Balazy, Klaudia and Tabor, Jacek},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2020},
pages = {263-278},
doi = {10.1007/978-3-030-67664-3_16},
url = {https://mlanthology.org/ecmlpkdd/2020/wojcik2020ecmlpkdd-finding/}
}