On the Surprising Effectiveness of a Single Global Merging in Decentralized Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910167943086080 |
|---|---|
| author | Zhu, Tongtian Zhang, Tianyu Wang, Mingze Zhou, Zhanpeng Wang, Can |
| author_facet | Zhu, Tongtian Zhang, Tianyu Wang, Mingze Zhou, Zhanpeng Wang, Can |
| contents | Decentralized learning provides a scalable alternative to parameter-server-based training, yet its performance is often hindered by limited peer-to-peer communication. In this paper, we study how communication should be scheduled over time, including determining when and how frequently devices synchronize. Counterintuitive empirical results show that concentrating communication budgets in the later stages of decentralized training remarkably improves global test performance. Surprisingly, we uncover that fully connected communication at the final step, implemented by a single global merging, can significantly improve the performance of decentralized learning under high data heterogeneity. Our theoretical contributions, which explain these phenomena, are the first to establish that the globally merged model of decentralized SGD can match the convergence rate of parallel SGD. Technically, we reinterpret part of the discrepancy among local models, which were previously considered as detrimental noise, as constructive components essential for matching this rate. This work provides evidence that decentralized learning is able to generalize under high data heterogeneity and limited communication, while offering broad new avenues for model merging research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_06542 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Surprising Effectiveness of a Single Global Merging in Decentralized Learning Zhu, Tongtian Zhang, Tianyu Wang, Mingze Zhou, Zhanpeng Wang, Can Machine Learning Distributed, Parallel, and Cluster Computing Multiagent Systems Decentralized learning provides a scalable alternative to parameter-server-based training, yet its performance is often hindered by limited peer-to-peer communication. In this paper, we study how communication should be scheduled over time, including determining when and how frequently devices synchronize. Counterintuitive empirical results show that concentrating communication budgets in the later stages of decentralized training remarkably improves global test performance. Surprisingly, we uncover that fully connected communication at the final step, implemented by a single global merging, can significantly improve the performance of decentralized learning under high data heterogeneity. Our theoretical contributions, which explain these phenomena, are the first to establish that the globally merged model of decentralized SGD can match the convergence rate of parallel SGD. Technically, we reinterpret part of the discrepancy among local models, which were previously considered as detrimental noise, as constructive components essential for matching this rate. This work provides evidence that decentralized learning is able to generalize under high data heterogeneity and limited communication, while offering broad new avenues for model merging research. |
| title | On the Surprising Effectiveness of a Single Global Merging in Decentralized Learning |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing Multiagent Systems |
| url | https://arxiv.org/abs/2507.06542 |