Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912524973113344 |
|---|---|
| author | Liu, Wei Panda, Anweshit Pandey, Ujwal Brissette, Christopher Shen, Yikang Slota, George M. Wang, Naigang Chen, Jie Xu, Yangyang |
| author_facet | Liu, Wei Panda, Anweshit Pandey, Ujwal Brissette, Christopher Shen, Yikang Slota, George M. Wang, Naigang Chen, Jie Xu, Yangyang |
| contents | In this paper, we design two compressed decentralized algorithms for solving nonconvex stochastic optimization under two different scenarios. Both algorithms adopt a momentum technique to achieve fast convergence and a message-compression technique to save communication costs. Though momentum acceleration and compressed communication have been used in literature, it is highly nontrivial to theoretically prove the effectiveness of their composition in a decentralized algorithm that can maintain the benefits of both sides, because of the need to simultaneously control the consensus error, the compression error, and the bias from the momentum gradient.
For the scenario where gradients are bounded, our proposal is a compressed decentralized adaptive method. To the best of our knowledge, this is the first decentralized adaptive stochastic gradient method with compressed communication. For the scenario of data heterogeneity without bounded gradients, our proposal is a compressed decentralized heavy-ball method, which applies a gradient tracking technique to address the challenge of data heterogeneity. Notably, both methods achieve an optimal convergence rate, and they can achieve linear speed up and adopt topology-independent algorithmic parameters within a certain regime of the user-specified error tolerance. Superior empirical performance is observed over state-of-the-art methods on training deep neural networks (DNNs) and Transformers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_04950 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization Liu, Wei Panda, Anweshit Pandey, Ujwal Brissette, Christopher Shen, Yikang Slota, George M. Wang, Naigang Chen, Jie Xu, Yangyang Machine Learning Optimization and Control In this paper, we design two compressed decentralized algorithms for solving nonconvex stochastic optimization under two different scenarios. Both algorithms adopt a momentum technique to achieve fast convergence and a message-compression technique to save communication costs. Though momentum acceleration and compressed communication have been used in literature, it is highly nontrivial to theoretically prove the effectiveness of their composition in a decentralized algorithm that can maintain the benefits of both sides, because of the need to simultaneously control the consensus error, the compression error, and the bias from the momentum gradient. For the scenario where gradients are bounded, our proposal is a compressed decentralized adaptive method. To the best of our knowledge, this is the first decentralized adaptive stochastic gradient method with compressed communication. For the scenario of data heterogeneity without bounded gradients, our proposal is a compressed decentralized heavy-ball method, which applies a gradient tracking technique to address the challenge of data heterogeneity. Notably, both methods achieve an optimal convergence rate, and they can achieve linear speed up and adopt topology-independent algorithmic parameters within a certain regime of the user-specified error tolerance. Superior empirical performance is observed over state-of-the-art methods on training deep neural networks (DNNs) and Transformers. |
| title | Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2508.04950 |