Batch-Expansion Training: An Efficient Optimization Framework
Abstract
We propose Batch-Expansion Training (BET), a framework for running a batch optimizer on a gradually expanding dataset. As opposed to stochastic approaches, batches do not need to be resampled i.i.d. at every iteration, thus making BET more resource efficient in a distributed setting, and when disk-access is constrained. Moreover, BET can be easily paired with most batch optimizers, does not require any parameter-tuning, and compares favorably to existing stochastic and batch methods. We show that when the batch size grows exponentially with the number of outer iterations, BET achieves optimal $O(1/\epsilon)$ data-access convergence rate for strongly convex objectives. Experiments in parallel and distributed settings show that BET performs better than standard batch and stochastic approaches.
Cite
Text
Derezinski et al. "Batch-Expansion Training: An Efficient Optimization Framework." International Conference on Artificial Intelligence and Statistics, 2018.Markdown
[Derezinski et al. "Batch-Expansion Training: An Efficient Optimization Framework." International Conference on Artificial Intelligence and Statistics, 2018.](https://mlanthology.org/aistats/2018/derezinski2018aistats-batch/)BibTeX
@inproceedings{derezinski2018aistats-batch,
title = {{Batch-Expansion Training: An Efficient Optimization Framework}},
author = {Derezinski, Michal and Mahajan, Dhruv and Keerthi, S. Sathiya and Vishwanathan, S. V. N. and Weimer, Markus},
booktitle = {International Conference on Artificial Intelligence and Statistics},
year = {2018},
pages = {736-744},
url = {https://mlanthology.org/aistats/2018/derezinski2018aistats-batch/}
}