ML Anthology

Training Discrete-Valued Neural Networks with Sign Activations Using Weight Distributions

Wolfgang Roth, Günther Schindler, Holger Fröning, Franz Pernkopf
ECML-PKDD 2019 pp. 382-398
doi:10.1007/978-3-030-46147-8_23 /ecmlpkdd/2019/roth2019ecmlpkdd-training/

Abstract

Since resource-constrained devices hardly benefit from the trend towards ever-increasing neural network (NN) structures, there is growing interest in designing more hardware-friendly NNs. In this paper, we consider the training of NNs with discrete-valued weights and sign activation functions that can be implemented more efficiently in terms of inference speed, memory requirements, and power consumption. We build on the framework of probabilistic forward propagations using the local reparameterization trick, where instead of training a single set of NN weights we rather train a distribution over these weights. Using this approach, we can perform gradient-based learning by optimizing the continuous distribution parameters over discrete weights while at the same time perform backpropagation through the sign activation. In our experiments, we investigate the influence of the number of weights on the classification performance on several benchmark datasets, and we show that our method achieves state-of-the-art performance.

PDF ECML-PKDD Semantic Scholar

Cite

Text

Roth et al. "Training Discrete-Valued Neural Networks with Sign Activations Using Weight Distributions." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2019. doi:10.1007/978-3-030-46147-8_23

Markdown

[Roth et al. "Training Discrete-Valued Neural Networks with Sign Activations Using Weight Distributions." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2019.](https://mlanthology.org/ecmlpkdd/2019/roth2019ecmlpkdd-training/) doi:10.1007/978-3-030-46147-8_23

BibTeX

@inproceedings{roth2019ecmlpkdd-training,
  title     = {{Training Discrete-Valued Neural Networks with Sign Activations Using Weight Distributions}},
  author    = {Roth, Wolfgang and Schindler, Günther and Fröning, Holger and Pernkopf, Franz},
  booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
  year      = {2019},
  pages     = {382-398},
  doi       = {10.1007/978-3-030-46147-8_23},
  url       = {https://mlanthology.org/ecmlpkdd/2019/roth2019ecmlpkdd-training/}
}