Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910016861110272 |
|---|---|
| author | Amer, Hossam Karim, Rezaul Pourranjbar, Ali Zhang, Weiwei Ahmed, Walid Chen, Boxing |
| author_facet | Amer, Hossam Karim, Rezaul Pourranjbar, Ali Zhang, Weiwei Ahmed, Walid Chen, Boxing |
| contents | With the rapid growth of large language models (LLMs), a wide range of methods have been developed to distribute computation and memory across hardware devices for efficient training and inference. While existing surveys provide descriptive overviews of these techniques, systematic analysis of their benefits and trade offs and how such insights can inform principled methodology for designing optimal distributed systems remain limited. This paper offers a comprehensive review of collective operations and distributed parallel strategies, complemented by mathematical formulations to deepen theoretical understanding. We further examine hybrid parallelization designs, emphasizing communication computation overlap across different stages of model deployment, including both training and inference. Recent advances in automated search for optimal hybrid parallelization strategies using cost models are also discussed. Moreover, we present case studies with mainstream architecture categories to reveal empirical insights to guide researchers and practitioners in parallelism strategy selection. Finally, we highlight open challenges and limitations of current LLM training paradigms and outline promising directions for the next generation of large scale model development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_09109 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide Amer, Hossam Karim, Rezaul Pourranjbar, Ali Zhang, Weiwei Ahmed, Walid Chen, Boxing Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Distributed, Parallel, and Cluster Computing With the rapid growth of large language models (LLMs), a wide range of methods have been developed to distribute computation and memory across hardware devices for efficient training and inference. While existing surveys provide descriptive overviews of these techniques, systematic analysis of their benefits and trade offs and how such insights can inform principled methodology for designing optimal distributed systems remain limited. This paper offers a comprehensive review of collective operations and distributed parallel strategies, complemented by mathematical formulations to deepen theoretical understanding. We further examine hybrid parallelization designs, emphasizing communication computation overlap across different stages of model deployment, including both training and inference. Recent advances in automated search for optimal hybrid parallelization strategies using cost models are also discussed. Moreover, we present case studies with mainstream architecture categories to reveal empirical insights to guide researchers and practitioners in parallelism strategy selection. Finally, we highlight open challenges and limitations of current LLM training paradigms and outline promising directions for the next generation of large scale model development. |
| title | Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2602.09109 |