BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912393455468544 |
|---|---|
| author | Wang, Yuxin Chen, Yuhan Li, Zeyu Kang, Xueze Fang, Yuchu Zhou, Yeju Zheng, Yang Tang, Zhenheng He, Xin Guo, Rui Wang, Xin Wang, Qiang Zhou, Amelie Chi Chu, Xiaowen |
| author_facet | Wang, Yuxin Chen, Yuhan Li, Zeyu Kang, Xueze Fang, Yuchu Zhou, Yeju Zheng, Yang Tang, Zhenheng He, Xin Guo, Rui Wang, Xin Wang, Qiang Zhou, Amelie Chi Chu, Xiaowen |
| contents | Serving systems for Large Language Models (LLMs) are often optimized to improve quality of service (QoS) and throughput. However, due to the lack of open-source LLM serving workloads, these systems are frequently evaluated under unrealistic workload assumptions. Consequently, performance may degrade when systems are deployed in real-world scenarios. This work presents BurstGPT, an LLM serving workload with 10.31 million traces from regional Azure OpenAI GPT services over 213 days. BurstGPT captures LLM serving characteristics from user, model and system perspectives: (1) User request concurrency: burstiness variations of requests in Azure OpenAI GPT services, revealing diversified concurrency patterns in different services and model types. (2) User conversation patterns: counts and intervals within conversations for service optimizations. (3) Model response lengths: auto-regressive serving processes of GPT models, showing statistical relations between requests and their responses. (4) System response failures: failures of conversation and API services, showing intensive resource needs and limited availability of LLM services in Azure. The details of the characteristics can serve multiple purposes in LLM serving optimizations, such as system evaluation and trace provisioning. In our demo evaluation with BurstGPT, frequent variations in BurstGPT reveal declines in efficiency, stability, or reliability in realistic LLM serving. We identify that the generalization of KV cache management, scheduling and disaggregation optimizations can be improved under realistic workload evaluations. BurstGPT is publicly available now at https://github.com/HPMLL/BurstGPT and is widely used to develop prototypes of LLM serving frameworks in the industry. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_17644 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems Wang, Yuxin Chen, Yuhan Li, Zeyu Kang, Xueze Fang, Yuchu Zhou, Yeju Zheng, Yang Tang, Zhenheng He, Xin Guo, Rui Wang, Xin Wang, Qiang Zhou, Amelie Chi Chu, Xiaowen Distributed, Parallel, and Cluster Computing Performance Serving systems for Large Language Models (LLMs) are often optimized to improve quality of service (QoS) and throughput. However, due to the lack of open-source LLM serving workloads, these systems are frequently evaluated under unrealistic workload assumptions. Consequently, performance may degrade when systems are deployed in real-world scenarios. This work presents BurstGPT, an LLM serving workload with 10.31 million traces from regional Azure OpenAI GPT services over 213 days. BurstGPT captures LLM serving characteristics from user, model and system perspectives: (1) User request concurrency: burstiness variations of requests in Azure OpenAI GPT services, revealing diversified concurrency patterns in different services and model types. (2) User conversation patterns: counts and intervals within conversations for service optimizations. (3) Model response lengths: auto-regressive serving processes of GPT models, showing statistical relations between requests and their responses. (4) System response failures: failures of conversation and API services, showing intensive resource needs and limited availability of LLM services in Azure. The details of the characteristics can serve multiple purposes in LLM serving optimizations, such as system evaluation and trace provisioning. In our demo evaluation with BurstGPT, frequent variations in BurstGPT reveal declines in efficiency, stability, or reliability in realistic LLM serving. We identify that the generalization of KV cache management, scheduling and disaggregation optimizations can be improved under realistic workload evaluations. BurstGPT is publicly available now at https://github.com/HPMLL/BurstGPT and is widely used to develop prototypes of LLM serving frameworks in the industry. |
| title | BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems |
| topic | Distributed, Parallel, and Cluster Computing Performance |
| url | https://arxiv.org/abs/2401.17644 |