Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.09486 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910526336925696 |
|---|---|
| author | Huang, Tao Chen, Pengfei Gong, Kyoka Hawk, Jocky Bright, Zachary Xie, Wenxin Huang, Kecheng Ji, Zhi |
| author_facet | Huang, Tao Chen, Pengfei Gong, Kyoka Hawk, Jocky Bright, Zachary Xie, Wenxin Huang, Kecheng Ji, Zhi |
| contents | Since the increasing popularity of large language model (LLM) backend systems, it is common and necessary to deploy stable serverless serving of LLM on multi-GPU clusters with autoscaling. However, there exist challenges because the diversity and co-location of applications in multi-GPU clusters will lead to low service quality and GPU utilization. To address them, we build ENOVA, a deployment, monitoring and autoscaling service towards serverless LLM serving. ENOVA deconstructs the execution process of LLM service comprehensively, based on which ENOVA designs a configuration recommendation module for automatic deployment on any GPU clusters and a performance detection module for autoscaling. On top of them, ENOVA implements a deployment execution engine for multi-GPU cluster scheduling. The experiment results show that ENOVA significantly outperforms other state-of-the-art methods and is suitable for wide deployment in large online systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09486 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Serving Huang, Tao Chen, Pengfei Gong, Kyoka Hawk, Jocky Bright, Zachary Xie, Wenxin Huang, Kecheng Ji, Zhi Distributed, Parallel, and Cluster Computing Artificial Intelligence Since the increasing popularity of large language model (LLM) backend systems, it is common and necessary to deploy stable serverless serving of LLM on multi-GPU clusters with autoscaling. However, there exist challenges because the diversity and co-location of applications in multi-GPU clusters will lead to low service quality and GPU utilization. To address them, we build ENOVA, a deployment, monitoring and autoscaling service towards serverless LLM serving. ENOVA deconstructs the execution process of LLM service comprehensively, based on which ENOVA designs a configuration recommendation module for automatic deployment on any GPU clusters and a performance detection module for autoscaling. On top of them, ENOVA implements a deployment execution engine for multi-GPU cluster scheduling. The experiment results show that ENOVA significantly outperforms other state-of-the-art methods and is suitable for wide deployment in large online systems. |
| title | ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Serving |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2407.09486 |