Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2412.04964 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915058041225216 |
|---|---|
| author | Li, Qingyuan Zhang, Bo Ye, Liang Zhang, Yifan Wu, Wei Sun, Yerui Ma, Lin Xie, Yuchen |
| author_facet | Li, Qingyuan Zhang, Bo Ye, Liang Zhang, Yifan Wu, Wei Sun, Yerui Ma, Lin Xie, Yuchen |
| contents | The ever-increasing sizes of large language models necessitate distributed solutions for fast inference that exploit multi-dimensional parallelism, where computational loads are split across various accelerators such as GPU clusters. However, this approach often introduces significant communication overhead, especially on devices with limited bandwidth. In this paper, we introduce Flash Communication, a novel low-bit compression technique designed to alleviate the tensor-parallelism communication bottleneck during inference. Our method substantially boosts intra-node communication speed by more than 3x and reduces the time-to-first-token by 2x, with nearly no sacrifice in model accuracy. Extensive experiments on various up-to-date LLMs demonstrate the effectiveness of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04964 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference Li, Qingyuan Zhang, Bo Ye, Liang Zhang, Yifan Wu, Wei Sun, Yerui Ma, Lin Xie, Yuchen Artificial Intelligence The ever-increasing sizes of large language models necessitate distributed solutions for fast inference that exploit multi-dimensional parallelism, where computational loads are split across various accelerators such as GPU clusters. However, this approach often introduces significant communication overhead, especially on devices with limited bandwidth. In this paper, we introduce Flash Communication, a novel low-bit compression technique designed to alleviate the tensor-parallelism communication bottleneck during inference. Our method substantially boosts intra-node communication speed by more than 3x and reduces the time-to-first-token by 2x, with nearly no sacrifice in model accuracy. Extensive experiments on various up-to-date LLMs demonstrate the effectiveness of our approach. |
| title | Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2412.04964 |