Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916458864312320 |
|---|---|
| author | Han, Wenchen Vargaftik, Shay Mitzenmacher, Michael Karp, Brad Basat, Ran Ben |
| author_facet | Han, Wenchen Vargaftik, Shay Mitzenmacher, Michael Karp, Brad Basat, Ran Ben |
| contents | Gradient aggregation has long been identified as a major bottleneck in today's large-scale distributed machine learning training systems. One promising solution to mitigate such bottlenecks is gradient compression, directly reducing communicated gradient data volume. However, in practice, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy.
In this work, we identify common issues in previous gradient compression systems and evaluation methodologies. These include excessive computational overheads; incompatibility with all-reduce; and insufficient evaluation methods, such as not using an end-to-end metric or using a 32-bit baseline instead of the stronger 16-bit baseline. We revisit common compression approaches (sparsification, quantization, and low-rank decomposition) and demonstrate how considering the above issues can lead to minor but strategic design changes, resulting in notably better performance. Our goal is to raise awareness of the need for design and evaluation standards that naturally translate to the end-to-end utility of gradient compression. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01378 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression Han, Wenchen Vargaftik, Shay Mitzenmacher, Michael Karp, Brad Basat, Ran Ben Machine Learning Networking and Internet Architecture Gradient aggregation has long been identified as a major bottleneck in today's large-scale distributed machine learning training systems. One promising solution to mitigate such bottlenecks is gradient compression, directly reducing communicated gradient data volume. However, in practice, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy. In this work, we identify common issues in previous gradient compression systems and evaluation methodologies. These include excessive computational overheads; incompatibility with all-reduce; and insufficient evaluation methods, such as not using an end-to-end metric or using a 32-bit baseline instead of the stronger 16-bit baseline. We revisit common compression approaches (sparsification, quantization, and low-rank decomposition) and demonstrate how considering the above issues can lead to minor but strategic design changes, resulting in notably better performance. Our goal is to raise awareness of the need for design and evaluation standards that naturally translate to the end-to-end utility of gradient compression. |
| title | Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression |
| topic | Machine Learning Networking and Internet Architecture |
| url | https://arxiv.org/abs/2407.01378 |