Using Supervised Pretraining to Improve Generalization of Neural Networks on Binary Classification Problems
Abstract
Neural networks are known to be very sensitive to the initial weights. There has been a lot of research on initialization that aims to stabilize the training process. However, very little research has studied the relationship between initialization and generalization. We demonstrate that poorly initialized model will lead to lower test accuracy. We propose a supervised pretraining technique that helps improve generalization on binary classification problems. The experimental results on four UCI datasets show that the proposed pretraining leads to higher test accuracy compared to the he_normal initialization when the training set is small. In further experiments on synthetic data, the improvement on test accuracy using the proposed pretraining reaches more than 30% when the data has high dimensionality and noisy features. Code related to this paper is available at: https://github.com/superRookie007/supervised_pretraining .
Cite
Text
Peng et al. "Using Supervised Pretraining to Improve Generalization of Neural Networks on Binary Classification Problems." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2018. doi:10.1007/978-3-030-10925-7_25Markdown
[Peng et al. "Using Supervised Pretraining to Improve Generalization of Neural Networks on Binary Classification Problems." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2018.](https://mlanthology.org/ecmlpkdd/2018/peng2018ecmlpkdd-using/) doi:10.1007/978-3-030-10925-7_25BibTeX
@inproceedings{peng2018ecmlpkdd-using,
title = {{Using Supervised Pretraining to Improve Generalization of Neural Networks on Binary Classification Problems}},
author = {Peng, Alex Yuxuan and Koh, Yun Sing and Riddle, Patricia and Pfahringer, Bernhard},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2018},
pages = {410-425},
doi = {10.1007/978-3-030-10925-7_25},
url = {https://mlanthology.org/ecmlpkdd/2018/peng2018ecmlpkdd-using/}
}