Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2603.29235 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911556487348224 |
|---|---|
| author | Zheng, Yusheng Mao, Wenan Cheng, Shuyi Feng, Fuqiu Li, Guangshui Liao, Zhaoyan Huang, Yongzhuo Xiao, Zhenwei Li, Yuqing Quinn, Andi Ma, Tao |
| author_facet | Zheng, Yusheng Mao, Wenan Cheng, Shuyi Feng, Fuqiu Li, Guangshui Liao, Zhaoyan Huang, Yongzhuo Xiao, Zhenwei Li, Yuqing Quinn, Andi Ma, Tao |
| contents | Performance diagnosis in production-scale AI training is challenging because subtle OS-level issues can trigger cascading GPU delays and network slowdowns, degrading training efficiency across thousands of GPUs. Existing profiling tools are limited to single system layers, incur prohibitive overhead (10--30%), or lack continuous deployment capabilities, resulting in manual analyses spanning days. We argue that continuous, cross-layer observability enabled by OS-level instrumentation and layered differential diagnosis is necessary to address this gap. We introduce SysOM-AI, a production observability system that continuously integrates CPU stack profiling, GPU kernel tracing, and NCCL event instrumentation via adaptive hybrid stack unwinding and eBPF-based tracing, incurring less than 0.4% overhead. Deployed at Alibaba across over 80,000 GPUs for more than one year, SysOM-AI helped diagnose 94 confirmed production issues, reducing median diagnosis time from days to approximately 10 minutes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_29235 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SysOM-AI: Continuous Cross-Layer Performance Diagnosis for Production AI Training Zheng, Yusheng Mao, Wenan Cheng, Shuyi Feng, Fuqiu Li, Guangshui Liao, Zhaoyan Huang, Yongzhuo Xiao, Zhenwei Li, Yuqing Quinn, Andi Ma, Tao Performance Performance diagnosis in production-scale AI training is challenging because subtle OS-level issues can trigger cascading GPU delays and network slowdowns, degrading training efficiency across thousands of GPUs. Existing profiling tools are limited to single system layers, incur prohibitive overhead (10--30%), or lack continuous deployment capabilities, resulting in manual analyses spanning days. We argue that continuous, cross-layer observability enabled by OS-level instrumentation and layered differential diagnosis is necessary to address this gap. We introduce SysOM-AI, a production observability system that continuously integrates CPU stack profiling, GPU kernel tracing, and NCCL event instrumentation via adaptive hybrid stack unwinding and eBPF-based tracing, incurring less than 0.4% overhead. Deployed at Alibaba across over 80,000 GPUs for more than one year, SysOM-AI helped diagnose 94 confirmed production issues, reducing median diagnosis time from days to approximately 10 minutes. |
| title | SysOM-AI: Continuous Cross-Layer Performance Diagnosis for Production AI Training |
| topic | Performance |
| url | https://arxiv.org/abs/2603.29235 |