Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.07529 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913231477407744 |
|---|---|
| author | Li, Haoyu Xu, Yuchen Chen, Jiayi Dwivedula, Rohit Wu, Wenfei He, Keqiang Akella, Aditya Kim, Daehyeok |
| author_facet | Li, Haoyu Xu, Yuchen Chen, Jiayi Dwivedula, Rohit Wu, Wenfei He, Keqiang Akella, Aditya Kim, Daehyeok |
| contents | As deep neural networks (DNNs) grow in complexity and size, the resultant increase in communication overhead during distributed training has become a significant bottleneck, challenging the scalability of distributed training systems. Existing solutions, while aiming to mitigate this bottleneck through worker-level compression and in-network aggregation, fall short due to their inability to efficiently reconcile the trade-offs between compression effectiveness and computational overhead, hindering overall performance and scalability. In this paper, we introduce a novel compression algorithm that effectively merges worker-level compression with in-network aggregation. Our solution is both homomorphic, allowing for efficient in-network aggregation without CPU/GPU processing, and lossless, ensuring no compromise on training accuracy. Theoretically optimal in compression and computational efficiency, our approach is empirically validated across diverse DNN models such as NCF, LSTM, VGG19, and BERT-base, showing up to a 6.33$\times$ improvement in aggregation throughput and a 3.74$\times$ increase in per-iteration training speed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_07529 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Accelerating Distributed Deep Learning using Lossless Homomorphic Compression Li, Haoyu Xu, Yuchen Chen, Jiayi Dwivedula, Rohit Wu, Wenfei He, Keqiang Akella, Aditya Kim, Daehyeok Distributed, Parallel, and Cluster Computing Data Structures and Algorithms Machine Learning Networking and Internet Architecture As deep neural networks (DNNs) grow in complexity and size, the resultant increase in communication overhead during distributed training has become a significant bottleneck, challenging the scalability of distributed training systems. Existing solutions, while aiming to mitigate this bottleneck through worker-level compression and in-network aggregation, fall short due to their inability to efficiently reconcile the trade-offs between compression effectiveness and computational overhead, hindering overall performance and scalability. In this paper, we introduce a novel compression algorithm that effectively merges worker-level compression with in-network aggregation. Our solution is both homomorphic, allowing for efficient in-network aggregation without CPU/GPU processing, and lossless, ensuring no compromise on training accuracy. Theoretically optimal in compression and computational efficiency, our approach is empirically validated across diverse DNN models such as NCF, LSTM, VGG19, and BERT-base, showing up to a 6.33$\times$ improvement in aggregation throughput and a 3.74$\times$ increase in per-iteration training speed. |
| title | Accelerating Distributed Deep Learning using Lossless Homomorphic Compression |
| topic | Distributed, Parallel, and Cluster Computing Data Structures and Algorithms Machine Learning Networking and Internet Architecture |
| url | https://arxiv.org/abs/2402.07529 |