Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918484215070720 |
|---|---|
| author | Chen, Jiu Yang, Shuangyan Xiong, Xu Duan, Hexiao Zhang, Xinran Ren, Jie Li, Dong |
| author_facet | Chen, Jiu Yang, Shuangyan Xiong, Xu Duan, Hexiao Zhang, Xinran Ren, Jie Li, Dong |
| contents | Decentralized LLM inference distributes computation among heterogeneous nodes across the internet, offering a performant and cost-efficient solution, alternative to traditional centralized inference. However, the low cross-node network bandwidth makes communication the primary bottleneck. In this paper, we introduce BloomBee, an internet-scale distributed LLM inference framework. BloomBee integrates LLM-layer assignment, micro-batching and tensor offloading to optimize communication from multiple dimensions. Additionally, BloomBee formulates the coordination of these techniques as an optimization problem and solves it using dynamic programming. BloomBee also customizes lossless compression and speculative decoding according to low-bandwidth network settings to reduce communication overhead. We evaluate BloomBee across a spectrum of network environments and show that it improves service throughput by up to 1.76x. It also reduces average latency by up to 43.20% compared to state-of-the-art decentralized LLM inference systems. BloomBee is open-sourced. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21072 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization Chen, Jiu Yang, Shuangyan Xiong, Xu Duan, Hexiao Zhang, Xinran Ren, Jie Li, Dong Distributed, Parallel, and Cluster Computing Decentralized LLM inference distributes computation among heterogeneous nodes across the internet, offering a performant and cost-efficient solution, alternative to traditional centralized inference. However, the low cross-node network bandwidth makes communication the primary bottleneck. In this paper, we introduce BloomBee, an internet-scale distributed LLM inference framework. BloomBee integrates LLM-layer assignment, micro-batching and tensor offloading to optimize communication from multiple dimensions. Additionally, BloomBee formulates the coordination of these techniques as an optimization problem and solves it using dynamic programming. BloomBee also customizes lossless compression and speculative decoding according to low-bandwidth network settings to reduce communication overhead. We evaluate BloomBee across a spectrum of network environments and show that it improves service throughput by up to 1.76x. It also reduces average latency by up to 43.20% compared to state-of-the-art decentralized LLM inference systems. BloomBee is open-sourced. |
| title | Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2604.21072 |