CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916682368286720 |
|---|---|
| author | Suo, Jiashun Liao, Xiaojian Xiao, Limin Ruan, Li Wang, Jinquan Su, Xiao Huo, Zhisheng |
| author_facet | Suo, Jiashun Liao, Xiaojian Xiao, Limin Ruan, Li Wang, Jinquan Su, Xiao Huo, Zhisheng |
| contents | Large language models like GPT-4 are resource-intensive, but recent advancements suggest that smaller, specialized experts can outperform the monolithic models on specific tasks. The Collaboration-of-Experts (CoE) approach integrates multiple expert models, improving the accuracy of generated results and offering great potential for precision-critical applications, such as automatic circuit board quality inspection. However, deploying CoE serving systems presents challenges to memory capacity due to the large number of experts required, which can lead to significant performance overhead from frequent expert switching across different memory and storage tiers.
We propose CoServe, an efficient CoE model serving system on heterogeneous CPU and GPU with limited memory. CoServe reduces unnecessary expert switching by leveraging expert dependency, a key property of CoE inference. CoServe introduces a dependency-aware request scheduler and dependency-aware expert management for efficient inference. It also introduces an offline profiler to automatically find optimal resource allocation on various processors and devices. In real-world intelligent manufacturing workloads, CoServe achieves 4.5$\times$ to 12$\times$ higher throughput compared to state-of-the-art systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02354 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory Suo, Jiashun Liao, Xiaojian Xiao, Limin Ruan, Li Wang, Jinquan Su, Xiao Huo, Zhisheng Distributed, Parallel, and Cluster Computing Artificial Intelligence Performance Large language models like GPT-4 are resource-intensive, but recent advancements suggest that smaller, specialized experts can outperform the monolithic models on specific tasks. The Collaboration-of-Experts (CoE) approach integrates multiple expert models, improving the accuracy of generated results and offering great potential for precision-critical applications, such as automatic circuit board quality inspection. However, deploying CoE serving systems presents challenges to memory capacity due to the large number of experts required, which can lead to significant performance overhead from frequent expert switching across different memory and storage tiers. We propose CoServe, an efficient CoE model serving system on heterogeneous CPU and GPU with limited memory. CoServe reduces unnecessary expert switching by leveraging expert dependency, a key property of CoE inference. CoServe introduces a dependency-aware request scheduler and dependency-aware expert management for efficient inference. It also introduces an offline profiler to automatically find optimal resource allocation on various processors and devices. In real-world intelligent manufacturing workloads, CoServe achieves 4.5$\times$ to 12$\times$ higher throughput compared to state-of-the-art systems. |
| title | CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Performance |
| url | https://arxiv.org/abs/2503.02354 |