Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913720231264256 |
|---|---|
| author | Mei, Yixuan Zhuang, Yonghao Miao, Xupeng Yang, Juncheng Jia, Zhihao Vinayak, Rashmi |
| author_facet | Mei, Yixuan Zhuang, Yonghao Miao, Xupeng Yang, Juncheng Jia, Zhihao Vinayak, Rashmi |
| contents | This paper introduces Helix, a distributed system for high-throughput, low-latency large language model (LLM) serving in heterogeneous GPU clusters. The key idea behind Helix is to formulate inference computation of LLMs over heterogeneous GPUs and network connections as a max-flow problem on directed, weighted graphs, whose nodes represent GPU instances and edges capture both GPU and network heterogeneity through their capacities. Helix then uses a mixed integer linear programming (MILP) algorithm to discover highly optimized strategies to serve LLMs on heterogeneous GPUs. This approach allows Helix to jointly optimize model placement and request scheduling, two highly entangled tasks in heterogeneous LLM serving. Our evaluation on several heterogeneous clusters ranging from 24 to 42 GPU nodes shows that Helix improves serving throughput by up to 3.3x and reduces prompting and decoding latency by up to 66% and 24%, respectively, compared to existing approaches. Helix is available at https://github.com/Thesys-lab/Helix-ASPLOS25. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01566 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow Mei, Yixuan Zhuang, Yonghao Miao, Xupeng Yang, Juncheng Jia, Zhihao Vinayak, Rashmi Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning This paper introduces Helix, a distributed system for high-throughput, low-latency large language model (LLM) serving in heterogeneous GPU clusters. The key idea behind Helix is to formulate inference computation of LLMs over heterogeneous GPUs and network connections as a max-flow problem on directed, weighted graphs, whose nodes represent GPU instances and edges capture both GPU and network heterogeneity through their capacities. Helix then uses a mixed integer linear programming (MILP) algorithm to discover highly optimized strategies to serve LLMs on heterogeneous GPUs. This approach allows Helix to jointly optimize model placement and request scheduling, two highly entangled tasks in heterogeneous LLM serving. Our evaluation on several heterogeneous clusters ranging from 24 to 42 GPU nodes shows that Helix improves serving throughput by up to 3.3x and reduces prompting and decoding latency by up to 66% and 24%, respectively, compared to existing approaches. Helix is available at https://github.com/Thesys-lab/Helix-ASPLOS25. |
| title | Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow |
| topic | Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.01566 |