THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910354475319296 |
|---|---|
| author | Li, Minghao Basat, Ran Ben Vargaftik, Shay Lao, ChonLam Xu, Kevin Mitzenmacher, Michael Yu, Minlan |
| author_facet | Li, Minghao Basat, Ran Ben Vargaftik, Shay Lao, ChonLam Xu, Kevin Mitzenmacher, Michael Yu, Minlan |
| contents | Deep neural networks (DNNs) are the de facto standard for essential use cases, such as image classification, computer vision, and natural language processing. As DNNs and datasets get larger, they require distributed training on increasingly larger clusters. A main bottleneck is the resulting communication overhead where workers exchange model updates (i.e., gradients) on a per-round basis. To address this bottleneck and accelerate training, a widely-deployed approach is compression. However, previous deployments often apply bi-directional compression schemes by simply using a uni-directional gradient compression scheme in each direction. This results in significant computational overheads at the parameter server and increased compression error, leading to longer training and lower accuracy. We introduce Tensor Homomorphic Compression (THC), a novel bi-directional compression framework that enables the direct aggregation of compressed values and thus eliminating the aforementioned computational overheads. Moreover, THC is compatible with in-network aggregation (INA), which allows for further acceleration. Our evaluation shows that training representative vision and language models with THC reaches target accuracy by 1.40x to 1.47x faster using INA and 1.28x to 1.33x faster using a software PS compared with state-of-the-art systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_08545 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression Li, Minghao Basat, Ran Ben Vargaftik, Shay Lao, ChonLam Xu, Kevin Mitzenmacher, Michael Yu, Minlan Machine Learning Artificial Intelligence Networking and Internet Architecture Deep neural networks (DNNs) are the de facto standard for essential use cases, such as image classification, computer vision, and natural language processing. As DNNs and datasets get larger, they require distributed training on increasingly larger clusters. A main bottleneck is the resulting communication overhead where workers exchange model updates (i.e., gradients) on a per-round basis. To address this bottleneck and accelerate training, a widely-deployed approach is compression. However, previous deployments often apply bi-directional compression schemes by simply using a uni-directional gradient compression scheme in each direction. This results in significant computational overheads at the parameter server and increased compression error, leading to longer training and lower accuracy. We introduce Tensor Homomorphic Compression (THC), a novel bi-directional compression framework that enables the direct aggregation of compressed values and thus eliminating the aforementioned computational overheads. Moreover, THC is compatible with in-network aggregation (INA), which allows for further acceleration. Our evaluation shows that training representative vision and language models with THC reaches target accuracy by 1.40x to 1.47x faster using INA and 1.28x to 1.33x faster using a software PS compared with state-of-the-art systems. |
| title | THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression |
| topic | Machine Learning Artificial Intelligence Networking and Internet Architecture |
| url | https://arxiv.org/abs/2302.08545 |