ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912593266868224 |
|---|---|
| author | Zhao, Liangyu Maleki, Saeed Wang, Yuanhong Wang, Zezhou Yang, Ziyue Pourreza, Hossein Krishnamurthy, Arvind |
| author_facet | Zhao, Liangyu Maleki, Saeed Wang, Yuanhong Wang, Zezhou Yang, Ziyue Pourreza, Hossein Krishnamurthy, Arvind |
| contents | As modern DNN models grow ever larger, collective communications between the accelerators (allreduce, etc.) emerge as a significant performance bottleneck. Designing efficient communication schedules is challenging, given today's heterogeneous and diverse network fabrics. We present ForestColl, a tool that generates throughput-optimal schedules for any network topology. ForestColl constructs broadcast/aggregation spanning trees as the communication schedule, achieving theoretical optimality. Its schedule generation runs in polynomial time and is highly scalable. ForestColl supports any network fabric, including both switching fabrics and direct accelerator connections. We evaluated ForestColl on AMD MI250 and NVIDIA DGX A100 & H100 clusters. ForestColl showed significant improvements over the vendors' own optimized communication libraries across various settings and in LLM training. ForestColl also outperformed other state-of-the-art schedule generation techniques with both more efficient generated schedules and substantially faster generation speed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_06787 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics Zhao, Liangyu Maleki, Saeed Wang, Yuanhong Wang, Zezhou Yang, Ziyue Pourreza, Hossein Krishnamurthy, Arvind Networking and Internet Architecture Distributed, Parallel, and Cluster Computing Machine Learning As modern DNN models grow ever larger, collective communications between the accelerators (allreduce, etc.) emerge as a significant performance bottleneck. Designing efficient communication schedules is challenging, given today's heterogeneous and diverse network fabrics. We present ForestColl, a tool that generates throughput-optimal schedules for any network topology. ForestColl constructs broadcast/aggregation spanning trees as the communication schedule, achieving theoretical optimality. Its schedule generation runs in polynomial time and is highly scalable. ForestColl supports any network fabric, including both switching fabrics and direct accelerator connections. We evaluated ForestColl on AMD MI250 and NVIDIA DGX A100 & H100 clusters. ForestColl showed significant improvements over the vendors' own optimized communication libraries across various settings and in LLM training. ForestColl also outperformed other state-of-the-art schedule generation techniques with both more efficient generated schedules and substantially faster generation speed. |
| title | ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics |
| topic | Networking and Internet Architecture Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2402.06787 |