Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2512.24750 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908741353340928 |
|---|---|
| author | Li, Wenxue Liu, Xiangzhou Li, Yuxuan Jin, Yilun Ren, Zhenghang Liao, Xudong Tian, Han Ren, Bo Zhong, Zhizhen Liu, Guyue Zhang, Ying Chen, Kai |
| author_facet | Li, Wenxue Liu, Xiangzhou Li, Yuxuan Jin, Yilun Ren, Zhenghang Liao, Xudong Tian, Han Ren, Bo Zhong, Zhizhen Liu, Guyue Zhang, Ying Chen, Kai |
| contents | Effective communication is essential in distributed training, with predictability being one of its most significant characteristics. However, existing studies primarily focus on exploiting predictability through online profiling for runtime optimization, without a systematic understanding of it. In this work, we aim to systematically formulate communication predictability in distributed training, particularly in Large Language Models (LLMs) that utilize hybrid parallelism. Our analysis focuses on both traffic patterns and communication overhead. Specifically, we investigate predictable traffic patterns in typical LLMs and evaluate how various factors influence GPU utilization and effective bandwidth (two critical variables affecting communication overhead). Furthermore, we develop an analytical formulation to estimate communication overhead in LLM training, which is validated with high accuracy against empirical data. Leveraging this formulation, we propose a configuration tuning tool, ConfigTuner, to optimize training performance. Compared to Megatron-LM, the training configurations optimized by ConfigTuner demonstrate up to a 1.36$\times$ increase in throughput. Compared to Alpa, ConfigTuner generates the same configuration suggestion while significantly reducing the search complexity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_24750 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Analyzing Communication Predictability in LLM Training Li, Wenxue Liu, Xiangzhou Li, Yuxuan Jin, Yilun Ren, Zhenghang Liao, Xudong Tian, Han Ren, Bo Zhong, Zhizhen Liu, Guyue Zhang, Ying Chen, Kai Networking and Internet Architecture Effective communication is essential in distributed training, with predictability being one of its most significant characteristics. However, existing studies primarily focus on exploiting predictability through online profiling for runtime optimization, without a systematic understanding of it. In this work, we aim to systematically formulate communication predictability in distributed training, particularly in Large Language Models (LLMs) that utilize hybrid parallelism. Our analysis focuses on both traffic patterns and communication overhead. Specifically, we investigate predictable traffic patterns in typical LLMs and evaluate how various factors influence GPU utilization and effective bandwidth (two critical variables affecting communication overhead). Furthermore, we develop an analytical formulation to estimate communication overhead in LLM training, which is validated with high accuracy against empirical data. Leveraging this formulation, we propose a configuration tuning tool, ConfigTuner, to optimize training performance. Compared to Megatron-LM, the training configurations optimized by ConfigTuner demonstrate up to a 1.36$\times$ increase in throughput. Compared to Alpa, ConfigTuner generates the same configuration suggestion while significantly reducing the search complexity. |
| title | Analyzing Communication Predictability in LLM Training |
| topic | Networking and Internet Architecture |
| url | https://arxiv.org/abs/2512.24750 |