Astra: Efficient and Money-saving Automatic Parallel Strategies Search on Heterogeneous GPUs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910835571425280 |
|---|---|
| author | Wang, Peiran Li, Haibing Haohan, Fu Li, Shiyong Wang, Yanpeng Shen, Dou |
| author_facet | Wang, Peiran Li, Haibing Haohan, Fu Li, Shiyong Wang, Yanpeng Shen, Dou |
| contents | In this paper, we introduce an efficient and money-saving automatic parallel strategies search framework on heterogeneous GPUs: Astra. First, Astra searches for the efficiency-optimal parallel strategy in both GPU configurations search space (GPU types and GPU numbers) and parallel parameters search space. Then, Astra also provides the solution on heterogeneous GPUs by mathematically modeling the time consumption of heterogeneous training. At last, Astra is the first to propose the automatic parallel strategy search on money-saving. The experiment results demonstrate that Astra can achieve better throughput than expert-designed strategies. The search time cost for Astra can also be limited to 1.27 seconds in a single-GPU setting and less than 1.35 minutes in a heterogeneous-GPU setting on average with an accuracy of over 95%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13480 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Astra: Efficient and Money-saving Automatic Parallel Strategies Search on Heterogeneous GPUs Wang, Peiran Li, Haibing Haohan, Fu Li, Shiyong Wang, Yanpeng Shen, Dou Distributed, Parallel, and Cluster Computing Artificial Intelligence In this paper, we introduce an efficient and money-saving automatic parallel strategies search framework on heterogeneous GPUs: Astra. First, Astra searches for the efficiency-optimal parallel strategy in both GPU configurations search space (GPU types and GPU numbers) and parallel parameters search space. Then, Astra also provides the solution on heterogeneous GPUs by mathematically modeling the time consumption of heterogeneous training. At last, Astra is the first to propose the automatic parallel strategy search on money-saving. The experiment results demonstrate that Astra can achieve better throughput than expert-designed strategies. The search time cost for Astra can also be limited to 1.27 seconds in a single-GPU setting and less than 1.35 minutes in a heterogeneous-GPU setting on average with an accuracy of over 95%. |
| title | Astra: Efficient and Money-saving Automatic Parallel Strategies Search on Heterogeneous GPUs |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2502.13480 |