Faster LLM Inference using DBMS-Inspired Preemption and Cache Replacement Policies
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916983529799680 |
|---|---|
| author | Kim, Kyoungmin Li, Jiacheng Hong, Kijae Ailamaki, Anastasia |
| author_facet | Kim, Kyoungmin Li, Jiacheng Hong, Kijae Ailamaki, Anastasia |
| contents | LLMs are increasingly used world-wide from daily tasks to agentic systems and data analytics, requiring significant GPU resources. LLM inference systems, however, are slow compared to database systems, and inference performance and mechanism have been often regarded as a black box, limiting the expansion of the use of LLMs inside databases and other performance-critical applications. This paper first analyzes the LLM inference performance and focuses on a data management issue inside LLM inference. We find that inference systems lack an adequate resource cost model and optimization strategy to schedule requests with their intermediate results in a cache reside in GPU memory when executing multiple concurrent inference requests. We adapt classic database techniques by building cost models for concurrent inference requests and a new cache replacement policy tailored for LLM inference, which can substantially save GPU costs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07447 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Faster LLM Inference using DBMS-Inspired Preemption and Cache Replacement Policies Kim, Kyoungmin Li, Jiacheng Hong, Kijae Ailamaki, Anastasia Performance Artificial Intelligence LLMs are increasingly used world-wide from daily tasks to agentic systems and data analytics, requiring significant GPU resources. LLM inference systems, however, are slow compared to database systems, and inference performance and mechanism have been often regarded as a black box, limiting the expansion of the use of LLMs inside databases and other performance-critical applications. This paper first analyzes the LLM inference performance and focuses on a data management issue inside LLM inference. We find that inference systems lack an adequate resource cost model and optimization strategy to schedule requests with their intermediate results in a cache reside in GPU memory when executing multiple concurrent inference requests. We adapt classic database techniques by building cost models for concurrent inference requests and a new cache replacement policy tailored for LLM inference, which can substantially save GPU costs. |
| title | Faster LLM Inference using DBMS-Inspired Preemption and Cache Replacement Policies |
| topic | Performance Artificial Intelligence |
| url | https://arxiv.org/abs/2411.07447 |