Temporal Predictive Coding for Gradient Compression in Distributed Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929525613395968 |
|---|---|
| author | Edin, Adrian Chen, Zheng Kieffer, Michel Johansson, Mikael |
| author_facet | Edin, Adrian Chen, Zheng Kieffer, Michel Johansson, Mikael |
| contents | This paper proposes a prediction-based gradient compression method for distributed learning with event-triggered communication. Our goal is to reduce the amount of information transmitted from the distributed agents to the parameter server by exploiting temporal correlation in the local gradients. We use a linear predictor that \textit{combines past gradients to form a prediction of the current gradient}, with coefficients that are optimized by solving a least-square problem. In each iteration, every agent transmits the predictor coefficients to the server such that the predicted local gradient can be computed. The difference between the true local gradient and the predicted one, termed the \textit{prediction residual, is only transmitted when its norm is above some threshold.} When this additional communication step is omitted, the server uses the prediction as the estimated gradient. This proposed design shows notable performance gains compared to existing methods in the literature, achieving convergence with reduced communication costs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02478 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Temporal Predictive Coding for Gradient Compression in Distributed Learning Edin, Adrian Chen, Zheng Kieffer, Michel Johansson, Mikael Information Theory Distributed, Parallel, and Cluster Computing Machine Learning Signal Processing This paper proposes a prediction-based gradient compression method for distributed learning with event-triggered communication. Our goal is to reduce the amount of information transmitted from the distributed agents to the parameter server by exploiting temporal correlation in the local gradients. We use a linear predictor that \textit{combines past gradients to form a prediction of the current gradient}, with coefficients that are optimized by solving a least-square problem. In each iteration, every agent transmits the predictor coefficients to the server such that the predicted local gradient can be computed. The difference between the true local gradient and the predicted one, termed the \textit{prediction residual, is only transmitted when its norm is above some threshold.} When this additional communication step is omitted, the server uses the prediction as the estimated gradient. This proposed design shows notable performance gains compared to existing methods in the literature, achieving convergence with reduced communication costs. |
| title | Temporal Predictive Coding for Gradient Compression in Distributed Learning |
| topic | Information Theory Distributed, Parallel, and Cluster Computing Machine Learning Signal Processing |
| url | https://arxiv.org/abs/2410.02478 |