Beyond A Single AI Cluster: A Survey of Decentralized LLM Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915515524448256 |
|---|---|
| author | Dong, Haotian Jiang, Jingyan Lu, Rongwei Luo, Jiajun Song, Jiajun Li, Bowen Shen, Ying Wang, Zhi |
| author_facet | Dong, Haotian Jiang, Jingyan Lu, Rongwei Luo, Jiajun Song, Jiajun Li, Bowen Shen, Ying Wang, Zhi |
| contents | The emergence of large language models (LLMs) has revolutionized AI development, yet the resource demands beyond a single cluster or even datacenter, limiting accessibility to well-resourced organizations. Decentralized training has emerged as a promising paradigm to leverage dispersed resources across clusters, datacenters and regions, offering the potential to democratize LLM development for broader communities. As the first comprehensive exploration of this emerging field, we present decentralized LLM training as a resource-driven paradigm and categorize existing efforts into community-driven and organizational approaches. We further clarify this through: (1) a comparison with related paradigms, (2) a characterization of decentralized resources, and (3) a taxonomy of recent advancements. We also provide up-to-date case studies and outline future directions to advance research in decentralized LLM training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_11023 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond A Single AI Cluster: A Survey of Decentralized LLM Training Dong, Haotian Jiang, Jingyan Lu, Rongwei Luo, Jiajun Song, Jiajun Li, Bowen Shen, Ying Wang, Zhi Distributed, Parallel, and Cluster Computing The emergence of large language models (LLMs) has revolutionized AI development, yet the resource demands beyond a single cluster or even datacenter, limiting accessibility to well-resourced organizations. Decentralized training has emerged as a promising paradigm to leverage dispersed resources across clusters, datacenters and regions, offering the potential to democratize LLM development for broader communities. As the first comprehensive exploration of this emerging field, we present decentralized LLM training as a resource-driven paradigm and categorize existing efforts into community-driven and organizational approaches. We further clarify this through: (1) a comparison with related paradigms, (2) a characterization of decentralized resources, and (3) a taxonomy of recent advancements. We also provide up-to-date case studies and outline future directions to advance research in decentralized LLM training. |
| title | Beyond A Single AI Cluster: A Survey of Decentralized LLM Training |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2503.11023 |