Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908339980468224 |
|---|---|
| author | Wu, Tianyuan Cao, Lunxi Lu, Hanfeng Jiang, Xiaoxiao Yu, Yinghao Yang, Siran Yang, Guodong Wang, Jiamang Qu, Lin Zhang, Liping Wang, Wei |
| author_facet | Wu, Tianyuan Cao, Lunxi Lu, Hanfeng Jiang, Xiaoxiao Yu, Yinghao Yang, Siran Yang, Guodong Wang, Jiamang Qu, Lin Zhang, Liping Wang, Wei |
| contents | Training large Deep Neural Network (DNN) models at scale often encounters straggler issues, mostly in communications due to network congestion, RNIC/switch defects, or topological asymmetry. Under advanced pipeline parallelism, even minor communication delays can induce significant training slowdowns. This occurs because (1) slow communication disrupts the pipeline schedule, creating cascading "bubbles" in a domino effect, and (2) current GPU kernel scheduling is susceptible to head-of-line blocking, where slow communication blocks subsequent computations, further adding to these bubbles. To address these challenges, we present ADAPTRA, a straggler-resilient training system with two key optimizations. First, it optimally adapts the pipeline schedule in the presence of stragglers to absorb communication delays without inducing cascading bubbles, using a simple yet effective algorithm guided by an analytical model. Second, upon detecting slow communication, ADAPTRA offloads communication operations from GPU to host memory and utilizes CPU-side RDMA for data transfer. This eliminates head-of-line blocking as subsequent computation kernels can be scheduled immediately on GPUs. Together, these optimizations effectively reduce pipeline stalls in the presence of communication stragglers, improving the training iteration time by 1.2-3.5x in our experiments under various settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_19232 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation Wu, Tianyuan Cao, Lunxi Lu, Hanfeng Jiang, Xiaoxiao Yu, Yinghao Yang, Siran Yang, Guodong Wang, Jiamang Qu, Lin Zhang, Liping Wang, Wei Distributed, Parallel, and Cluster Computing Training large Deep Neural Network (DNN) models at scale often encounters straggler issues, mostly in communications due to network congestion, RNIC/switch defects, or topological asymmetry. Under advanced pipeline parallelism, even minor communication delays can induce significant training slowdowns. This occurs because (1) slow communication disrupts the pipeline schedule, creating cascading "bubbles" in a domino effect, and (2) current GPU kernel scheduling is susceptible to head-of-line blocking, where slow communication blocks subsequent computations, further adding to these bubbles. To address these challenges, we present ADAPTRA, a straggler-resilient training system with two key optimizations. First, it optimally adapts the pipeline schedule in the presence of stragglers to absorb communication delays without inducing cascading bubbles, using a simple yet effective algorithm guided by an analytical model. Second, upon detecting slow communication, ADAPTRA offloads communication operations from GPU to host memory and utilizes CPU-side RDMA for data transfer. This eliminates head-of-line blocking as subsequent computation kernels can be scheduled immediately on GPUs. Together, these optimizations effectively reduce pipeline stalls in the presence of communication stragglers, improving the training iteration time by 1.2-3.5x in our experiments under various settings. |
| title | Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2504.19232 |