FAST: An Efficient Scheduler for All-to-All GPU Communication
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915838740660224 |
|---|---|
| author | Lei, Yiran Lee, Dongjoo Zhao, Liangyu Kurniawan, Daniar Kim, Chanmyeong Jeong, Heetaek Kim, Changsu Choi, Hyeonseong Yu, Liangcheng Krishnamurthy, Arvind Sherry, Justine Nurvitadhi, Eriko |
| author_facet | Lei, Yiran Lee, Dongjoo Zhao, Liangyu Kurniawan, Daniar Kim, Chanmyeong Jeong, Heetaek Kim, Changsu Choi, Hyeonseong Yu, Liangcheng Krishnamurthy, Arvind Sherry, Justine Nurvitadhi, Eriko |
| contents | All-to-All(v) communication is a critical primitive in modern machine learning workloads, particularly mixture-of-experts (MoE) models. Unfortunately, efficient scheduling is challenging due to workload skew, heterogeneous two-tier fabrics, and incast congestion, compounded by the dynamic nature of MoE workloads, where traffic shifts every few hundred milliseconds. Existing schedulers are hardly scalable, incurring seconds to hours of synthesis time, making them impractical. We present FAST, an efficient All-to-All(v) scheduler. FAST addresses skew through intra-server rebalancing and enforces balanced, one-to-one scale-out transfers that avoid incast. Evaluated extensively on both NVIDIA H200 and AMD MI300X clusters, FAST consistently outperforms state-of-the-art solutions on skewed workloads while reducing synthesis time by orders of magnitude. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_09764 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FAST: An Efficient Scheduler for All-to-All GPU Communication Lei, Yiran Lee, Dongjoo Zhao, Liangyu Kurniawan, Daniar Kim, Chanmyeong Jeong, Heetaek Kim, Changsu Choi, Hyeonseong Yu, Liangcheng Krishnamurthy, Arvind Sherry, Justine Nurvitadhi, Eriko Distributed, Parallel, and Cluster Computing Networking and Internet Architecture All-to-All(v) communication is a critical primitive in modern machine learning workloads, particularly mixture-of-experts (MoE) models. Unfortunately, efficient scheduling is challenging due to workload skew, heterogeneous two-tier fabrics, and incast congestion, compounded by the dynamic nature of MoE workloads, where traffic shifts every few hundred milliseconds. Existing schedulers are hardly scalable, incurring seconds to hours of synthesis time, making them impractical. We present FAST, an efficient All-to-All(v) scheduler. FAST addresses skew through intra-server rebalancing and enforces balanced, one-to-one scale-out transfers that avoid incast. Evaluated extensively on both NVIDIA H200 and AMD MI300X clusters, FAST consistently outperforms state-of-the-art solutions on skewed workloads while reducing synthesis time by orders of magnitude. |
| title | FAST: An Efficient Scheduler for All-to-All GPU Communication |
| topic | Distributed, Parallel, and Cluster Computing Networking and Internet Architecture |
| url | https://arxiv.org/abs/2505.09764 |