Data Split Strategiesfor Evolving Predictive Models
Abstract
A conventional textbook prescription for building good predictive models is to split the data into three parts: training set (for model fitting), validation set (for model selection), and test set (for final model assessment). Predictive models can potentially evolve over time as developers improve their performance either by acquiring new data or improving the existing model. The main contribution of this paper is to discuss problems encountered and propose workflows to manage the allocation of newly acquired data into different sets in such dynamic model building and updating scenarios. Specifically we propose three different workflows (parallel dump, serial waterfall, and hybrid) for allocating new data into the existing training, validation, and test splits. Particular emphasis is laid on avoiding the bias due to the repeated use of the existing validation or the test set.
Cite
Text
Raykar and Saha. "Data Split Strategiesfor Evolving Predictive Models." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2015. doi:10.1007/978-3-319-23528-8_1Markdown
[Raykar and Saha. "Data Split Strategiesfor Evolving Predictive Models." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2015.](https://mlanthology.org/ecmlpkdd/2015/raykar2015ecmlpkdd-data/) doi:10.1007/978-3-319-23528-8_1BibTeX
@inproceedings{raykar2015ecmlpkdd-data,
title = {{Data Split Strategiesfor Evolving Predictive Models}},
author = {Raykar, Vikas C. and Saha, Amrita},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2015},
pages = {3-19},
doi = {10.1007/978-3-319-23528-8_1},
url = {https://mlanthology.org/ecmlpkdd/2015/raykar2015ecmlpkdd-data/}
}