Dataset Efficient Training with Model Ensembling
Abstract
We propose a dataset-efficient deep learning training method by ensembling multiple models trained on different subsets. The ensembling method leverages the difficulty level of data samples to select subsets that are representative and diverse. The approach involves building a common base model with a random subset of data and then allotting different subsets to the models in an ensemble. The models are trained with their own subsets and then merged into a single model. We design an multi-phase training strategy that aggregates models in the ensemble more frequently and prevents divergence. The experiments on ResNet18 and ImageNet show that ensembling outperforms the no-ensemble case and achieves 64.8% accuracy with only 30% of dataset, saving 20 hours of training time in a single V100 GPU training experiment with a mild accuracy drop.
Cite
Text
Ro et al. "Dataset Efficient Training with Model Ensembling." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2023. doi:10.1109/CVPRW59228.2023.00497Markdown
[Ro et al. "Dataset Efficient Training with Model Ensembling." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2023.](https://mlanthology.org/cvprw/2023/ro2023cvprw-dataset/) doi:10.1109/CVPRW59228.2023.00497BibTeX
@inproceedings{ro2023cvprw-dataset,
title = {{Dataset Efficient Training with Model Ensembling}},
author = {Ro, Yeonju and Xu, Cong and Ciborowska, Agnieszka and Bhattacharya, Suparna and Li, Frankie and Foltin, Martin},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
year = {2023},
pages = {4700-4704},
doi = {10.1109/CVPRW59228.2023.00497},
url = {https://mlanthology.org/cvprw/2023/ro2023cvprw-dataset/}
}