The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909210907770880 |
|---|---|
| author | Zeng, Pai Ning, Zhenyu Zhao, Jieru Cui, Weihao Xu, Mengwei Guo, Liwei Chen, Xusheng Shan, Yizhou |
| author_facet | Zeng, Pai Ning, Zhenyu Zhao, Jieru Cui, Weihao Xu, Mengwei Guo, Liwei Chen, Xusheng Shan, Yizhou |
| contents | We survey the large language model (LLM) serving area to understand the intricate dynamics between cost-efficiency and accuracy, which is magnified by the growing need for longer contextual understanding when deploying models at a massive scale. Our findings reveal that works in this space optimize along three distinct but conflicting goals: improving serving context length (C), improving serving accuracy (A), and improving serving performance (P). Drawing inspiration from the CAP theorem in databases, we propose a CAP principle for LLM serving, which suggests that any optimization can improve at most two of these three goals simultaneously. Our survey categorizes existing works within this framework. We find the definition and continuity of user-perceived measurement metrics are crucial in determining whether a goal has been met, akin to prior CAP databases in the wild. We recognize the CAP principle for LLM serving as a guiding principle, rather than a formal theorem, to inform designers of the inherent and dynamic trade-offs in serving models. As serving accuracy and performance have been extensively studied, this survey focuses on works that extend serving context length and address the resulting challenges. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_11299 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving Zeng, Pai Ning, Zhenyu Zhao, Jieru Cui, Weihao Xu, Mengwei Guo, Liwei Chen, Xusheng Shan, Yizhou Databases Machine Learning We survey the large language model (LLM) serving area to understand the intricate dynamics between cost-efficiency and accuracy, which is magnified by the growing need for longer contextual understanding when deploying models at a massive scale. Our findings reveal that works in this space optimize along three distinct but conflicting goals: improving serving context length (C), improving serving accuracy (A), and improving serving performance (P). Drawing inspiration from the CAP theorem in databases, we propose a CAP principle for LLM serving, which suggests that any optimization can improve at most two of these three goals simultaneously. Our survey categorizes existing works within this framework. We find the definition and continuity of user-perceived measurement metrics are crucial in determining whether a goal has been met, akin to prior CAP databases in the wild. We recognize the CAP principle for LLM serving as a guiding principle, rather than a formal theorem, to inform designers of the inherent and dynamic trade-offs in serving models. As serving accuracy and performance have been extensively studied, this survey focuses on works that extend serving context length and address the resulting challenges. |
| title | The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving |
| topic | Databases Machine Learning |
| url | https://arxiv.org/abs/2405.11299 |