Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911136449822720 |
|---|---|
| author | Deng, Yangtao Zhang, Lei Wang, Qinlong Zhi, Xiaoyun Zhang, Xinlei Jiang, Zhuo Xu, Haohan Wang, Lei Song, Zuquan Liu, Gaohong Bai, Yang Wang, Shuguang Xiao, Wencong Ye, Jianxi Yu, Minlan Xu, Hong |
| author_facet | Deng, Yangtao Zhang, Lei Wang, Qinlong Zhi, Xiaoyun Zhang, Xinlei Jiang, Zhuo Xu, Haohan Wang, Lei Song, Zuquan Liu, Gaohong Bai, Yang Wang, Shuguang Xiao, Wencong Ye, Jianxi Yu, Minlan Xu, Hong |
| contents | Reliability is essential for ensuring efficiency in LLM training. However, many real-world reliability issues remain difficult to resolve, resulting in wasted resources and degraded model performance. Unfortunately, today's collective communication libraries operate as black boxes, hiding critical information needed for effective root cause analysis. We propose Mycroft, a lightweight distributed tracing and root cause analysis system designed to address previously hidden reliability issues in collective communication. Mycroft's key idea is to trace collective communication states and leverage internal control and data dependencies to resolve reliability problems in LLM training. Mycroft has been deployed at ByteDance for over six months to debug collective communication related issues at runtime. It detected anomalies within 15 seconds in 90% of cases and identified the root cause within 20 seconds in 60% of cases. We also conducted extensive fault injection experiments to demonstrate Mycroft's capability and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_03018 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training Deng, Yangtao Zhang, Lei Wang, Qinlong Zhi, Xiaoyun Zhang, Xinlei Jiang, Zhuo Xu, Haohan Wang, Lei Song, Zuquan Liu, Gaohong Bai, Yang Wang, Shuguang Xiao, Wencong Ye, Jianxi Yu, Minlan Xu, Hong Distributed, Parallel, and Cluster Computing Machine Learning Reliability is essential for ensuring efficiency in LLM training. However, many real-world reliability issues remain difficult to resolve, resulting in wasted resources and degraded model performance. Unfortunately, today's collective communication libraries operate as black boxes, hiding critical information needed for effective root cause analysis. We propose Mycroft, a lightweight distributed tracing and root cause analysis system designed to address previously hidden reliability issues in collective communication. Mycroft's key idea is to trace collective communication states and leverage internal control and data dependencies to resolve reliability problems in LLM training. Mycroft has been deployed at ByteDance for over six months to debug collective communication related issues at runtime. It detected anomalies within 15 seconds in 90% of cases and identified the root cause within 20 seconds in 60% of cases. We also conducted extensive fault injection experiments to demonstrate Mycroft's capability and efficiency. |
| title | Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2509.03018 |