Semi-Cyclic Stochastic Gradient Descent
Abstract
We consider convex SGD updates with a block-cyclic structure, i.e., where each cycle consists of a small number of blocks, each with many samples from a possibly different, block-specific, distribution. This situation arises, e.g., in Federated Learning where the mobile devices available for updates at different times during the day have different characteristics. We show that such block-cyclic structure can significantly deteriorate the performance of SGD, but propose a simple approach that allows prediction with the same guarantees as for i.i.d., non-cyclic, sampling.
Cite
Text
Eichner et al. "Semi-Cyclic Stochastic Gradient Descent." International Conference on Machine Learning, 2019.Markdown
[Eichner et al. "Semi-Cyclic Stochastic Gradient Descent." International Conference on Machine Learning, 2019.](https://mlanthology.org/icml/2019/eichner2019icml-semicyclic/)BibTeX
@inproceedings{eichner2019icml-semicyclic,
title = {{Semi-Cyclic Stochastic Gradient Descent}},
author = {Eichner, Hubert and Koren, Tomer and Mcmahan, Brendan and Srebro, Nathan and Talwar, Kunal},
booktitle = {International Conference on Machine Learning},
year = {2019},
pages = {1764-1773},
volume = {97},
url = {https://mlanthology.org/icml/2019/eichner2019icml-semicyclic/}
}