Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2311.05760 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929375672270848 |
|---|---|
| author | Campbell, Andrew Liu, Hang Woldemariam, Leah Scaglione, Anna |
| author_facet | Campbell, Andrew Liu, Hang Woldemariam, Leah Scaglione, Anna |
| contents | Recent research highlights frequent model communication as a significant bottleneck to the efficiency of decentralized machine learning (ML), especially for large-scale and over-parameterized neural networks (NNs). To address this, we present Malcom-PSGD, a novel decentralized ML algorithm that combines gradient compression techniques with model sparsification. We promote model sparsity by adding $\ell_1$ regularization to the objective and present a decentralized proximal SGD method for training. Our approach employs vector source coding and dithering-based quantization for the compressed gradient communication of sparsified models. Our analysis demonstrates that Malcom-PSGD achieves a convergence rate of $\mathcal{O}(1/\sqrt{t})$ with respect to the iterations $t$, assuming a constant consensus and learning rate. This result is supported by our proof for the convergence of non-convex compressed Proximal SGD methods. Additionally, we conduct a bit analysis, providing a closed-form expression for the communication costs associated with Malcom-PSGD. Numerical results verify our theoretical findings and demonstrate that our method reduces communication costs by approximately $75\%$ when compared to the state-of-the-art. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_05760 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Compressed and Sparse Models for Non-Convex Decentralized Learning Campbell, Andrew Liu, Hang Woldemariam, Leah Scaglione, Anna Machine Learning Distributed, Parallel, and Cluster Computing Data Structures and Algorithms Multiagent Systems Optimization and Control 68W15, 68W10, 68W40, 90C06, 90C35, 90C25 G.1.6; F.2.1; E.4 Recent research highlights frequent model communication as a significant bottleneck to the efficiency of decentralized machine learning (ML), especially for large-scale and over-parameterized neural networks (NNs). To address this, we present Malcom-PSGD, a novel decentralized ML algorithm that combines gradient compression techniques with model sparsification. We promote model sparsity by adding $\ell_1$ regularization to the objective and present a decentralized proximal SGD method for training. Our approach employs vector source coding and dithering-based quantization for the compressed gradient communication of sparsified models. Our analysis demonstrates that Malcom-PSGD achieves a convergence rate of $\mathcal{O}(1/\sqrt{t})$ with respect to the iterations $t$, assuming a constant consensus and learning rate. This result is supported by our proof for the convergence of non-convex compressed Proximal SGD methods. Additionally, we conduct a bit analysis, providing a closed-form expression for the communication costs associated with Malcom-PSGD. Numerical results verify our theoretical findings and demonstrate that our method reduces communication costs by approximately $75\%$ when compared to the state-of-the-art. |
| title | Compressed and Sparse Models for Non-Convex Decentralized Learning |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing Data Structures and Algorithms Multiagent Systems Optimization and Control 68W15, 68W10, 68W40, 90C06, 90C35, 90C25 G.1.6; F.2.1; E.4 |
| url | https://arxiv.org/abs/2311.05760 |