OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the Cloud
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912359119847424 |
|---|---|
| author | Warraich, Ertza Shabtai, Omer Manaa, Khalid Vargaftik, Shay Piasetzky, Yonatan Kadosh, Matty Suresh, Lalith Shahbaz, Muhammad |
| author_facet | Warraich, Ertza Shabtai, Omer Manaa, Khalid Vargaftik, Shay Piasetzky, Yonatan Kadosh, Matty Suresh, Lalith Shahbaz, Muhammad |
| contents | We present OptiReduce, a new collective-communication system for the cloud with bounded, predictable completion times for deep-learning jobs in the presence of varying computation (stragglers) and communication (congestion and gradient drops) variabilities. OptiReduce exploits the inherent resiliency and the stochastic nature of distributed deep-learning (DDL) training and fine-tuning to work with approximated (or lost) gradients -- providing an efficient balance between (tail) performance and the resulting accuracy of the trained models.
Exploiting this domain-specific characteristic of DDL, OptiReduce introduces (1) mechanisms (e.g., unreliable bounded transport with adaptive timeout) to improve the DDL jobs' tail execution time, and (2) strategies (e.g., Transpose AllReduce and Hadamard Transform) to mitigate the impact of gradient drops on model accuracy. Our evaluation shows that OptiReduce achieves 70% and 30% faster time-to-accuracy (TTA), on average, when operating in shared, cloud environments (e.g., CloudLab) compared to Gloo and NCCL, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_06993 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the Cloud Warraich, Ertza Shabtai, Omer Manaa, Khalid Vargaftik, Shay Piasetzky, Yonatan Kadosh, Matty Suresh, Lalith Shahbaz, Muhammad Distributed, Parallel, and Cluster Computing Networking and Internet Architecture We present OptiReduce, a new collective-communication system for the cloud with bounded, predictable completion times for deep-learning jobs in the presence of varying computation (stragglers) and communication (congestion and gradient drops) variabilities. OptiReduce exploits the inherent resiliency and the stochastic nature of distributed deep-learning (DDL) training and fine-tuning to work with approximated (or lost) gradients -- providing an efficient balance between (tail) performance and the resulting accuracy of the trained models. Exploiting this domain-specific characteristic of DDL, OptiReduce introduces (1) mechanisms (e.g., unreliable bounded transport with adaptive timeout) to improve the DDL jobs' tail execution time, and (2) strategies (e.g., Transpose AllReduce and Hadamard Transform) to mitigate the impact of gradient drops on model accuracy. Our evaluation shows that OptiReduce achieves 70% and 30% faster time-to-accuracy (TTA), on average, when operating in shared, cloud environments (e.g., CloudLab) compared to Gloo and NCCL, respectively. |
| title | OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the Cloud |
| topic | Distributed, Parallel, and Cluster Computing Networking and Internet Architecture |
| url | https://arxiv.org/abs/2310.06993 |