FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909361267277824 |
|---|---|
| author | Chang, Li-Wen Bao, Wenlei Hou, Qi Jiang, Chengquan Zheng, Ningxin Zhong, Yinmin Zhang, Xuanrun Song, Zuquan Yao, Chengji Jiang, Ziheng Lin, Haibin Jin, Xin Liu, Xin |
| author_facet | Chang, Li-Wen Bao, Wenlei Hou, Qi Jiang, Chengquan Zheng, Ningxin Zhong, Yinmin Zhang, Xuanrun Song, Zuquan Yao, Chengji Jiang, Ziheng Lin, Haibin Jin, Xin Liu, Xin |
| contents | Large deep learning models have demonstrated strong ability to solve many tasks across a wide range of applications. Those large models typically require training and inference to be distributed. Tensor parallelism is a common technique partitioning computation of an operation or layer across devices to overcome the memory capacity limitation of a single processor, and/or to accelerate computation to meet a certain latency requirement. However, this kind of parallelism introduces additional communication that might contribute a significant portion of overall runtime. Thus limits scalability of this technique within a group of devices with high speed interconnects, such as GPUs with NVLinks in a node. This paper proposes a novel method, Flux, to significantly hide communication latencies with dependent computations for GPUs. Flux over-decomposes communication and computation operations into much finer-grained operations and further fuses them into a larger kernel to effectively hide communication without compromising kernel efficiency. Flux can potentially overlap up to 96% of communication given a fused kernel. Overall, it can achieve up to 1.24x speedups for training over Megatron-LM on a cluster of 128 GPUs with various GPU generations and interconnects, and up to 1.66x and 1.30x speedups for prefill and decoding inference over vLLM on a cluster with 8 GPUs with various GPU generations and interconnects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06858 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion Chang, Li-Wen Bao, Wenlei Hou, Qi Jiang, Chengquan Zheng, Ningxin Zhong, Yinmin Zhang, Xuanrun Song, Zuquan Yao, Chengji Jiang, Ziheng Lin, Haibin Jin, Xin Liu, Xin Machine Learning Distributed, Parallel, and Cluster Computing Large deep learning models have demonstrated strong ability to solve many tasks across a wide range of applications. Those large models typically require training and inference to be distributed. Tensor parallelism is a common technique partitioning computation of an operation or layer across devices to overcome the memory capacity limitation of a single processor, and/or to accelerate computation to meet a certain latency requirement. However, this kind of parallelism introduces additional communication that might contribute a significant portion of overall runtime. Thus limits scalability of this technique within a group of devices with high speed interconnects, such as GPUs with NVLinks in a node. This paper proposes a novel method, Flux, to significantly hide communication latencies with dependent computations for GPUs. Flux over-decomposes communication and computation operations into much finer-grained operations and further fuses them into a larger kernel to effectively hide communication without compromising kernel efficiency. Flux can potentially overlap up to 96% of communication given a fused kernel. Overall, it can achieve up to 1.24x speedups for training over Megatron-LM on a cluster of 128 GPUs with various GPU generations and interconnects, and up to 1.66x and 1.30x speedups for prefill and decoding inference over vLLM on a cluster with 8 GPUs with various GPU generations and interconnects. |
| title | FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2406.06858 |