InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909078190555136 |
|---|---|
| author | Chen, Qiaoling Gu, Diandian Wang, Guoteng Chen, Xun Xiong, YingTong Huang, Ting Hu, Qinghao Jin, Xin Wen, Yonggang Zhang, Tianwei Sun, Peng |
| author_facet | Chen, Qiaoling Gu, Diandian Wang, Guoteng Chen, Xun Xiong, YingTong Huang, Ting Hu, Qinghao Jin, Xin Wen, Yonggang Zhang, Tianwei Sun, Peng |
| contents | Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are neither efficient nor compatible with commonly-used training algorithms such as FlashAttention. We design InternEvo to address these issues. InternEvo decouples all of the sharding dimensions into a new hierarchical space, and systematically analyzes the memory and communication cost of LLM training. Then, it generates an effective hybrid parallelism strategy. We design a new selective overlap mechanism to mitigate the communication overhead introduced by the hybrid parallelism. We also implement memory management techniques to reduce GPU memory fragmentation. Evaluation results show that InternEvo generates parallelization strategies that match or outperform existing methods in model FLOPs utilization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_09149 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding Chen, Qiaoling Gu, Diandian Wang, Guoteng Chen, Xun Xiong, YingTong Huang, Ting Hu, Qinghao Jin, Xin Wen, Yonggang Zhang, Tianwei Sun, Peng Distributed, Parallel, and Cluster Computing Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are neither efficient nor compatible with commonly-used training algorithms such as FlashAttention. We design InternEvo to address these issues. InternEvo decouples all of the sharding dimensions into a new hierarchical space, and systematically analyzes the memory and communication cost of LLM training. Then, it generates an effective hybrid parallelism strategy. We design a new selective overlap mechanism to mitigate the communication overhead introduced by the hybrid parallelism. We also implement memory management techniques to reduce GPU memory fragmentation. Evaluation results show that InternEvo generates parallelization strategies that match or outperform existing methods in model FLOPs utilization. |
| title | InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2401.09149 |