LASER: Linear Compression in Wireless Distributed Optimization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913224208678912 |
|---|---|
| author | Makkuva, Ashok Vardhan Bondaschi, Marco Vogels, Thijs Jaggi, Martin Kim, Hyeji Gastpar, Michael C. |
| author_facet | Makkuva, Ashok Vardhan Bondaschi, Marco Vogels, Thijs Jaggi, Martin Kim, Hyeji Gastpar, Michael C. |
| contents | Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its persistent issues. Most compression schemes to alleviate this either assume noiseless communication links, or fail to achieve good performance on practical tasks. In this paper, we close this gap and introduce LASER: LineAr CompreSsion in WirEless DistRibuted Optimization. LASER capitalizes on the inherent low-rank structure of gradients and transmits them efficiently over the noisy channels. Whilst enjoying theoretical guarantees similar to those of the classical SGD, LASER shows consistent gains over baselines on a variety of practical benchmarks. In particular, it outperforms the state-of-the-art compression schemes on challenging computer vision and GPT language modeling tasks. On the latter, we obtain $50$-$64 \%$ improvement in perplexity over our baselines for noisy channels. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_13033 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | LASER: Linear Compression in Wireless Distributed Optimization Makkuva, Ashok Vardhan Bondaschi, Marco Vogels, Thijs Jaggi, Martin Kim, Hyeji Gastpar, Michael C. Neural and Evolutionary Computing Artificial Intelligence Information Theory Machine Learning Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its persistent issues. Most compression schemes to alleviate this either assume noiseless communication links, or fail to achieve good performance on practical tasks. In this paper, we close this gap and introduce LASER: LineAr CompreSsion in WirEless DistRibuted Optimization. LASER capitalizes on the inherent low-rank structure of gradients and transmits them efficiently over the noisy channels. Whilst enjoying theoretical guarantees similar to those of the classical SGD, LASER shows consistent gains over baselines on a variety of practical benchmarks. In particular, it outperforms the state-of-the-art compression schemes on challenging computer vision and GPT language modeling tasks. On the latter, we obtain $50$-$64 \%$ improvement in perplexity over our baselines for noisy channels. |
| title | LASER: Linear Compression in Wireless Distributed Optimization |
| topic | Neural and Evolutionary Computing Artificial Intelligence Information Theory Machine Learning |
| url | https://arxiv.org/abs/2310.13033 |