Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918485820440576 |
|---|---|
| author | Mei, Yixuan Li, Zikun Chen, Zixuan Pan, Shiqi Wu, Mengdi Miao, Xupeng Jia, Zhihao Rashmi, K. V. |
| author_facet | Mei, Yixuan Li, Zikun Chen, Zixuan Pan, Shiqi Wu, Mengdi Miao, Xupeng Jia, Zhihao Rashmi, K. V. |
| contents | The usage of large language models (LLMs) has grown increasingly fragmented, with no single model dominating. Meanwhile, cloud providers offer a wide range of mid-tier and older-generation GPUs that enjoy better availability and deliver comparable performance per dollar to top-tier hardware. To efficiently harness these heterogeneous resources for serving multiple LLMs concurrently, we introduce Coral, an adaptive heterogeneity-aware multi-LLM serving system. The key idea behind Coral is to jointly optimize resource allocation and the serving strategy of each model replica across all models. To keep pace with shifting throughput demand and resource availability, Coral applies a lossless two-stage decomposition that preserves joint optimality while cutting online solve time from hours to tens of seconds. Our evaluation across 6 models and 20 GPU configurations shows that Coral reduces serving cost by up to 2.79$\times$ over the best baseline, and delivers up to 2.39$\times$ higher goodput under scarce resource availability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_04357 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs Mei, Yixuan Li, Zikun Chen, Zixuan Pan, Shiqi Wu, Mengdi Miao, Xupeng Jia, Zhihao Rashmi, K. V. Distributed, Parallel, and Cluster Computing Artificial Intelligence Computation and Language Machine Learning The usage of large language models (LLMs) has grown increasingly fragmented, with no single model dominating. Meanwhile, cloud providers offer a wide range of mid-tier and older-generation GPUs that enjoy better availability and deliver comparable performance per dollar to top-tier hardware. To efficiently harness these heterogeneous resources for serving multiple LLMs concurrently, we introduce Coral, an adaptive heterogeneity-aware multi-LLM serving system. The key idea behind Coral is to jointly optimize resource allocation and the serving strategy of each model replica across all models. To keep pace with shifting throughput demand and resource availability, Coral applies a lossless two-stage decomposition that preserves joint optimality while cutting online solve time from hours to tens of seconds. Our evaluation across 6 models and 20 GPU configurations shows that Coral reduces serving cost by up to 2.79$\times$ over the best baseline, and delivers up to 2.39$\times$ higher goodput under scarce resource availability. |
| title | Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2605.04357 |