Augment Your Batch: Improving Generalization Through Instance Repetition
Abstract
Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. We propose to use batch augmentation: replicating instances of samples within the same batch with different data augmentations. Batch augmentation acts as a regularizer and an accelerator, increasing both generalization and performance scaling for a fixed budget of optimization steps. We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of networks and datasets. Our results show that batch augmentation reduces the number of necessary SGD updates to achieve the same accuracy as the state-of-the-art. Overall, this simple yet effective method enables faster training and better generalization by allowing more computational resources to be used concurrently.
Cite
Text
Hoffer et al. "Augment Your Batch: Improving Generalization Through Instance Repetition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. doi:10.1109/CVPR42600.2020.00815Markdown
[Hoffer et al. "Augment Your Batch: Improving Generalization Through Instance Repetition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.](https://mlanthology.org/cvpr/2020/hoffer2020cvpr-augment/) doi:10.1109/CVPR42600.2020.00815BibTeX
@inproceedings{hoffer2020cvpr-augment,
title = {{Augment Your Batch: Improving Generalization Through Instance Repetition}},
author = {Hoffer, Elad and Ben-Nun, Tal and Hubara, Itay and Giladi, Niv and Hoefler, Torsten and Soudry, Daniel},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2020},
doi = {10.1109/CVPR42600.2020.00815},
url = {https://mlanthology.org/cvpr/2020/hoffer2020cvpr-augment/}
}