B-Ary Tree Push-Pull Method Is Provably Efficient for Distributed Learning on Heterogeneous Data

Abstract

This paper considers the distributed learning problem where a group of agents cooperatively minimizes the summation of their local cost functions based on peer-to-peer communication. Particularly, we propose a highly efficient algorithm, termed ``B-ary Tree Push-Pull'' (BTPP), that employs two B-ary spanning trees for distributing the information related to the parameters and stochastic gradients across the network. The simple method is efficient in communication since each agent interacts with at most $(B+1)$ neighbors per iteration. More importantly, BTPP achieves linear speedup for smooth nonconvex objective functions with only $\tilde{O}(n)$ transient iterations, significantly outperforming the state-of-the-art results to the best of our knowledge.

PDF NeurIPS OpenReview Semantic Scholar

Cite

Text

You and Pu. "B-Ary Tree Push-Pull Method Is Provably Efficient for Distributed Learning on Heterogeneous Data." Neural Information Processing Systems, 2024. doi:10.52202/079017-3094

Markdown

[You and Pu. "B-Ary Tree Push-Pull Method Is Provably Efficient for Distributed Learning on Heterogeneous Data." Neural Information Processing Systems, 2024.](https://mlanthology.org/neurips/2024/you2024neurips-bary/) doi:10.52202/079017-3094

BibTeX

@inproceedings{you2024neurips-bary,
  title     = {{B-Ary Tree Push-Pull Method Is Provably Efficient for Distributed Learning on Heterogeneous Data}},
  author    = {You, Runze and Pu, Shi},
  booktitle = {Neural Information Processing Systems},
  year      = {2024},
  doi       = {10.52202/079017-3094},
  url       = {https://mlanthology.org/neurips/2024/you2024neurips-bary/}
}