FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915429782388736 |
|---|---|
| author | Li, Qingyuan Zhang, Bo Kang, Hui Xu, Tianhao Qian, Yulei Xie, Yuchen Ma, Lin |
| author_facet | Li, Qingyuan Zhang, Bo Kang, Hui Xu, Tianhao Qian, Yulei Xie, Yuchen Ma, Lin |
| contents | Nowadays, communication bottlenecks have emerged as a critical challenge in the distributed training and deployment of large language models (LLMs). This paper introduces FlashCommunication V2, a novel communication paradigm enabling efficient cross-GPU transmission at arbitrary bit widths. Its core innovations lie in the proposed bit splitting and spike reserving techniques, which address the challenges of low-bit quantization. Bit splitting decomposes irregular bit widths into basic units, ensuring compatibility with hardware capabilities and thus enabling transmission at any bit width. Spike reserving, on the other hand, retains numerical outliers (i.e., minima and maxima) as floating-point numbers, which shrinks the dynamic numerical range and pushes the quantization limits to 2-bit with acceptable losses. FlashCommunication V2 significantly enhances the flexibility and resource utilization of communication systems. Through meticulous software-hardware co-design, it delivers robust performance and reduced overhead across both NVLink-based and PCIe-based architectures, achieving a maximum 3.2$\times$ speedup in AllReduce and 2$\times$ in All2All communication. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_03760 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication Li, Qingyuan Zhang, Bo Kang, Hui Xu, Tianhao Qian, Yulei Xie, Yuchen Ma, Lin Distributed, Parallel, and Cluster Computing Artificial Intelligence Nowadays, communication bottlenecks have emerged as a critical challenge in the distributed training and deployment of large language models (LLMs). This paper introduces FlashCommunication V2, a novel communication paradigm enabling efficient cross-GPU transmission at arbitrary bit widths. Its core innovations lie in the proposed bit splitting and spike reserving techniques, which address the challenges of low-bit quantization. Bit splitting decomposes irregular bit widths into basic units, ensuring compatibility with hardware capabilities and thus enabling transmission at any bit width. Spike reserving, on the other hand, retains numerical outliers (i.e., minima and maxima) as floating-point numbers, which shrinks the dynamic numerical range and pushes the quantization limits to 2-bit with acceptable losses. FlashCommunication V2 significantly enhances the flexibility and resource utilization of communication systems. Through meticulous software-hardware co-design, it delivers robust performance and reduced overhead across both NVLink-based and PCIe-based architectures, achieving a maximum 3.2$\times$ speedup in AllReduce and 2$\times$ in All2All communication. |
| title | FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2508.03760 |