InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910463646760960 |
|---|---|
| author | Abhyankar, Reyna He, Zijian Srivatsa, Vikranth Zhang, Hao Zhang, Yiying |
| author_facet | Abhyankar, Reyna He, Zijian Srivatsa, Vikranth Zhang, Hao Zhang, Yiying |
| contents | Large language models are increasingly integrated with external environments, tools, and agents like ChatGPT plugins to extend their capability beyond language-centric tasks. However, today's LLM inference systems are designed for standalone LLMs. They treat each external interaction as the end of LLM generation and form a new request when the interaction finishes, causing unnecessary recomputation of already computed contexts, which accounts for 37-40% of total model forwarding time. This paper presents InferCept, the first LLM inference framework targeting augmented LLMs and supporting the efficient interception of LLM generation. InferCept minimizes the GPU resource waste caused by LLM interceptions and dedicates saved memory for serving more requests. InferCept improves the overall serving throughput by 1.6x-2x and completes 2x more requests per second compared to the state-of-the-art LLM inference systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01869 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InferCept: Efficient Intercept Support for Augmented Large Language Model Inference Abhyankar, Reyna He, Zijian Srivatsa, Vikranth Zhang, Hao Zhang, Yiying Machine Learning Computation and Language Distributed, Parallel, and Cluster Computing Large language models are increasingly integrated with external environments, tools, and agents like ChatGPT plugins to extend their capability beyond language-centric tasks. However, today's LLM inference systems are designed for standalone LLMs. They treat each external interaction as the end of LLM generation and form a new request when the interaction finishes, causing unnecessary recomputation of already computed contexts, which accounts for 37-40% of total model forwarding time. This paper presents InferCept, the first LLM inference framework targeting augmented LLMs and supporting the efficient interception of LLM generation. InferCept minimizes the GPU resource waste caused by LLM interceptions and dedicates saved memory for serving more requests. InferCept improves the overall serving throughput by 1.6x-2x and completes 2x more requests per second compared to the state-of-the-art LLM inference systems. |
| title | InferCept: Efficient Intercept Support for Augmented Large Language Model Inference |
| topic | Machine Learning Computation and Language Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2402.01869 |