EROICA: Online Performance Troubleshooting for Large-scale Model Training
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908872661270528 |
|---|---|
| author | Guan, Yu Yin, Zhiyu Chen, Haoyu Cheng, Sheng Yang, Chaojie Qian, Kun Xu, Tianyin Zhang, Pengcheng Zhang, Yang Zhao, Hanyu Li, Yong Lin, Wei Cai, Dennis Zhai, Ennan |
| author_facet | Guan, Yu Yin, Zhiyu Chen, Haoyu Cheng, Sheng Yang, Chaojie Qian, Kun Xu, Tianyin Zhang, Pengcheng Zhang, Yang Zhao, Hanyu Li, Yong Lin, Wei Cai, Dennis Zhai, Ennan |
| contents | Troubleshooting performance problems of large model training (LMT) is immensely challenging, due to unprecedented scales of modern GPU clusters, the complexity of software-hardware interactions, and the data intensity of the training process. Existing troubleshooting approaches designed for traditional distributed systems or datacenter networks fall short and can hardly apply to real-world training systems. In this paper, we present EROICA, the first online troubleshooting system that provides both fine-grained observation based on profiling, and coverage of all machines in GPU clusters, to diagnose performance issues in production, including both hardware and software problems (or the mixture of both). EROICA effectively summarizes runtime behavior patterns of LMT function executions via online profiling, and leverages differential observability to localize the root cause with minimal production impact. EROICA has been deployed as a production service for large-scale GPU clusters of ~100,000 GPUs for 1.5 years. It has diagnosed a variety of difficult performance issues with 97.5% success. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08528 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EROICA: Online Performance Troubleshooting for Large-scale Model Training Guan, Yu Yin, Zhiyu Chen, Haoyu Cheng, Sheng Yang, Chaojie Qian, Kun Xu, Tianyin Zhang, Pengcheng Zhang, Yang Zhao, Hanyu Li, Yong Lin, Wei Cai, Dennis Zhai, Ennan Distributed, Parallel, and Cluster Computing Machine Learning Operating Systems Troubleshooting performance problems of large model training (LMT) is immensely challenging, due to unprecedented scales of modern GPU clusters, the complexity of software-hardware interactions, and the data intensity of the training process. Existing troubleshooting approaches designed for traditional distributed systems or datacenter networks fall short and can hardly apply to real-world training systems. In this paper, we present EROICA, the first online troubleshooting system that provides both fine-grained observation based on profiling, and coverage of all machines in GPU clusters, to diagnose performance issues in production, including both hardware and software problems (or the mixture of both). EROICA effectively summarizes runtime behavior patterns of LMT function executions via online profiling, and leverages differential observability to localize the root cause with minimal production impact. EROICA has been deployed as a production service for large-scale GPU clusters of ~100,000 GPUs for 1.5 years. It has diagnosed a variety of difficult performance issues with 97.5% success. |
| title | EROICA: Online Performance Troubleshooting for Large-scale Model Training |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning Operating Systems |
| url | https://arxiv.org/abs/2506.08528 |