SLowcalSGD : Slow Query Points Improve Local-SGD for Stochastic Convex Optimization

Abstract

We consider distributed learning scenarios where $M$ machines interact with a parameter server along several communication rounds in order to minimize a joint objective function. Focusing on the heterogeneous case, where different machines may draw samples from different data-distributions, we design the first local update method that provably benefits over the two most prominent distributed baselines: namely Minibatch-SGD and Local-SGD. Key to our approach is a slow querying technique that we customize to the distributed setting, which in turn enables a better mitigation of the bias caused by local updates.

PDF NeurIPS OpenReview Semantic Scholar

Cite

Text

Dahan and Levy. "SLowcalSGD : Slow Query Points Improve Local-SGD for Stochastic Convex Optimization." Neural Information Processing Systems, 2024. doi:10.52202/079017-2957

Markdown

[Dahan and Levy. "SLowcalSGD : Slow Query Points Improve Local-SGD for Stochastic Convex Optimization." Neural Information Processing Systems, 2024.](https://mlanthology.org/neurips/2024/dahan2024neurips-slowcalsgd/) doi:10.52202/079017-2957

BibTeX

@inproceedings{dahan2024neurips-slowcalsgd,
  title     = {{SLowcalSGD : Slow Query Points Improve Local-SGD for Stochastic Convex Optimization}},
  author    = {Dahan, Tehila and Levy, Kfir Y.},
  booktitle = {Neural Information Processing Systems},
  year      = {2024},
  doi       = {10.52202/079017-2957},
  url       = {https://mlanthology.org/neurips/2024/dahan2024neurips-slowcalsgd/}
}