B-ary Tree Push-Pull Method is Provably Efficient for Distributed Learning on Heterogeneous Data
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916489778429952 |
|---|---|
| author | You, Runze Pu, Shi |
| author_facet | You, Runze Pu, Shi |
| contents | This paper considers the distributed learning problem where a group of agents cooperatively minimizes the summation of their local cost functions based on peer-to-peer communication. Particularly, we propose a highly efficient algorithm, termed ``B-ary Tree Push-Pull'' (BTPP), that employs two B-ary spanning trees for distributing the information related to the parameters and stochastic gradients across the network. The simple method is efficient in communication since each agent interacts with at most $(B+1)$ neighbors per iteration. More importantly, BTPP achieves linear speedup for smooth nonconvex and strongly convex objective functions with only $\tilde{O}(n)$ and $\tilde{O}(1)$ transient iterations, respectively, significantly outperforming the state-of-the-art results to the best of our knowledge. Our code is available at https://github.com/ryou98/BTPP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_05454 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | B-ary Tree Push-Pull Method is Provably Efficient for Distributed Learning on Heterogeneous Data You, Runze Pu, Shi Optimization and Control This paper considers the distributed learning problem where a group of agents cooperatively minimizes the summation of their local cost functions based on peer-to-peer communication. Particularly, we propose a highly efficient algorithm, termed ``B-ary Tree Push-Pull'' (BTPP), that employs two B-ary spanning trees for distributing the information related to the parameters and stochastic gradients across the network. The simple method is efficient in communication since each agent interacts with at most $(B+1)$ neighbors per iteration. More importantly, BTPP achieves linear speedup for smooth nonconvex and strongly convex objective functions with only $\tilde{O}(n)$ and $\tilde{O}(1)$ transient iterations, respectively, significantly outperforming the state-of-the-art results to the best of our knowledge. Our code is available at https://github.com/ryou98/BTPP. |
| title | B-ary Tree Push-Pull Method is Provably Efficient for Distributed Learning on Heterogeneous Data |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2404.05454 |