Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.04272 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909331177340928 |
|---|---|
| author | Feng, Hao Zhang, Boyuan Ye, Fanjiang Si, Min Chu, Ching-Hsiang Tian, Jiannan Yin, Chunxing Deng, Summer Hao, Yuchen Balaji, Pavan Geng, Tong Tao, Dingwen |
| author_facet | Feng, Hao Zhang, Boyuan Ye, Fanjiang Si, Min Chu, Ching-Hsiang Tian, Jiannan Yin, Chunxing Deng, Summer Hao, Yuchen Balaji, Pavan Geng, Tong Tao, Dingwen |
| contents | DLRM is a state-of-the-art recommendation system model that has gained widespread adoption across various industry applications. The large size of DLRM models, however, necessitates the use of multiple devices/GPUs for efficient training. A significant bottleneck in this process is the time-consuming all-to-all communication required to collect embedding data from all devices. To mitigate this, we introduce a method that employs error-bounded lossy compression to reduce the communication data size and accelerate DLRM training. We develop a novel error-bounded lossy compression algorithm, informed by an in-depth analysis of embedding data features, to achieve high compression ratios. Moreover, we introduce a dual-level adaptive strategy for error-bound adjustment, spanning both table-wise and iteration-wise aspects, to balance the compression benefits with the potential impacts on accuracy. We further optimize our compressor for PyTorch tensors on GPUs, minimizing compression overhead. Evaluation shows that our method achieves a 1.38$\times$ training speedup with a minimal accuracy impact. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_04272 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compression Feng, Hao Zhang, Boyuan Ye, Fanjiang Si, Min Chu, Ching-Hsiang Tian, Jiannan Yin, Chunxing Deng, Summer Hao, Yuchen Balaji, Pavan Geng, Tong Tao, Dingwen Machine Learning Distributed, Parallel, and Cluster Computing DLRM is a state-of-the-art recommendation system model that has gained widespread adoption across various industry applications. The large size of DLRM models, however, necessitates the use of multiple devices/GPUs for efficient training. A significant bottleneck in this process is the time-consuming all-to-all communication required to collect embedding data from all devices. To mitigate this, we introduce a method that employs error-bounded lossy compression to reduce the communication data size and accelerate DLRM training. We develop a novel error-bounded lossy compression algorithm, informed by an in-depth analysis of embedding data features, to achieve high compression ratios. Moreover, we introduce a dual-level adaptive strategy for error-bound adjustment, spanning both table-wise and iteration-wise aspects, to balance the compression benefits with the potential impacts on accuracy. We further optimize our compressor for PyTorch tensors on GPUs, minimizing compression overhead. Evaluation shows that our method achieves a 1.38$\times$ training speedup with a minimal accuracy impact. |
| title | Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compression |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2407.04272 |