MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909769545023488 |
|---|---|
| author | Liao, Xudong Sun, Yijun Tian, Han Wan, Xinchen Jin, Yilun Wang, Zilong Ren, Zhenghang Huang, Xinyang Li, Wenxue Tse, Kin Fai Zhong, Zhizhen Liu, Guyue Zhang, Ying Ye, Xiaofeng Zhang, Yiming Chen, Kai |
| author_facet | Liao, Xudong Sun, Yijun Tian, Han Wan, Xinchen Jin, Yilun Wang, Zilong Ren, Zhenghang Huang, Xinyang Li, Wenxue Tse, Kin Fai Zhong, Zhizhen Liu, Guyue Zhang, Ying Ye, Xiaofeng Zhang, Yiming Chen, Kai |
| contents | Mixture-of-Expert (MoE) models outperform conventional models by selectively activating different subnets, named experts, on a per-token basis. This gated computation generates dynamic communications that cannot be determined beforehand, challenging the existing GPU interconnects that remain static during the distributed training process. In this paper, we advocate for a first-of-its-kind system, called MixNet, that unlocks topology reconfiguration during distributed MoE training. Towards this vision, we first perform a production measurement study and show that the MoE dynamic communication pattern has strong locality, alleviating the requirement of global reconfiguration. Based on this, we design and implement a regionally reconfigurable high-bandwidth domain on top of existing electrical interconnects using optical circuit switching (OCS), achieving scalability while maintaining rapid adaptability. We have built a fully functional MixNet prototype with commodity hardware and a customized collective communication runtime that trains state-of-the-art MoE models with in-training topology reconfiguration across 32 A100 GPUs. Large-scale packet-level simulations show that MixNet delivers comparable performance as the non-blocking fat-tree fabric while boosting the training cost efficiency (e.g., performance per dollar) of four representative MoE models by 1.2x-1.5x and 1.9x-2.3x at 100 Gbps and 400 Gbps link bandwidths, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_03905 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training Liao, Xudong Sun, Yijun Tian, Han Wan, Xinchen Jin, Yilun Wang, Zilong Ren, Zhenghang Huang, Xinyang Li, Wenxue Tse, Kin Fai Zhong, Zhizhen Liu, Guyue Zhang, Ying Ye, Xiaofeng Zhang, Yiming Chen, Kai Networking and Internet Architecture Machine Learning Mixture-of-Expert (MoE) models outperform conventional models by selectively activating different subnets, named experts, on a per-token basis. This gated computation generates dynamic communications that cannot be determined beforehand, challenging the existing GPU interconnects that remain static during the distributed training process. In this paper, we advocate for a first-of-its-kind system, called MixNet, that unlocks topology reconfiguration during distributed MoE training. Towards this vision, we first perform a production measurement study and show that the MoE dynamic communication pattern has strong locality, alleviating the requirement of global reconfiguration. Based on this, we design and implement a regionally reconfigurable high-bandwidth domain on top of existing electrical interconnects using optical circuit switching (OCS), achieving scalability while maintaining rapid adaptability. We have built a fully functional MixNet prototype with commodity hardware and a customized collective communication runtime that trains state-of-the-art MoE models with in-training topology reconfiguration across 32 A100 GPUs. Large-scale packet-level simulations show that MixNet delivers comparable performance as the non-blocking fat-tree fabric while boosting the training cost efficiency (e.g., performance per dollar) of four representative MoE models by 1.2x-1.5x and 1.9x-2.3x at 100 Gbps and 400 Gbps link bandwidths, respectively. |
| title | MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training |
| topic | Networking and Internet Architecture Machine Learning |
| url | https://arxiv.org/abs/2501.03905 |