Jenga: Effective Memory Management for Serving LLM with Heterogeneity
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909549984743424 |
|---|---|
| author | Zhang, Chen Du, Kuntai Liu, Shu Kwon, Woosuk Mo, Xiangxi Wang, Yufeng Liu, Xiaoxuan You, Kaichao Li, Zhuohan Long, Mingsheng Zhai, Jidong Gonzalez, Joseph Stoica, Ion |
| author_facet | Zhang, Chen Du, Kuntai Liu, Shu Kwon, Woosuk Mo, Xiangxi Wang, Yufeng Liu, Xiaoxuan You, Kaichao Li, Zhuohan Long, Mingsheng Zhai, Jidong Gonzalez, Joseph Stoica, Ion |
| contents | Large language models (LLMs) are widely used but expensive to run, especially as inference workloads grow. To lower costs, maximizing the request batch size by managing GPU memory efficiently is crucial. While PagedAttention has recently been proposed to improve the efficiency of memory management, we find that the growing heterogeneity in the embeddings dimensions, attention, and access patterns of modern LLM architectures introduces new challenges for memory allocation.
In this paper, we present Jenga, a novel memory allocation framework for heterogeneous embeddings in LLMs. Jenga tackles two key challenges: (1) minimizing memory fragmentation when managing embeddings of different sizes, and (2) enabling flexible caching and eviction policies tailored to the specific token-dependency patterns of various layers. Jenga employs a two-level memory allocator, leveraging the least common multiple (LCM) of embedding sizes to optimize memory usage and providing APIs to express layer-specific caching logic to enhance memory reuse.
We implemente Jenga on vLLM, a state-of-the-art LLM inference engine, and evaluate it with diverse LLMs, datasets, and GPU configurations. Evaluations show that Jenga improves GPU memory utilization by up to 79.6%, and increases serving throughput by up to 4.92x (1.80x on average). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_18292 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Jenga: Effective Memory Management for Serving LLM with Heterogeneity Zhang, Chen Du, Kuntai Liu, Shu Kwon, Woosuk Mo, Xiangxi Wang, Yufeng Liu, Xiaoxuan You, Kaichao Li, Zhuohan Long, Mingsheng Zhai, Jidong Gonzalez, Joseph Stoica, Ion Distributed, Parallel, and Cluster Computing Large language models (LLMs) are widely used but expensive to run, especially as inference workloads grow. To lower costs, maximizing the request batch size by managing GPU memory efficiently is crucial. While PagedAttention has recently been proposed to improve the efficiency of memory management, we find that the growing heterogeneity in the embeddings dimensions, attention, and access patterns of modern LLM architectures introduces new challenges for memory allocation. In this paper, we present Jenga, a novel memory allocation framework for heterogeneous embeddings in LLMs. Jenga tackles two key challenges: (1) minimizing memory fragmentation when managing embeddings of different sizes, and (2) enabling flexible caching and eviction policies tailored to the specific token-dependency patterns of various layers. Jenga employs a two-level memory allocator, leveraging the least common multiple (LCM) of embedding sizes to optimize memory usage and providing APIs to express layer-specific caching logic to enhance memory reuse. We implemente Jenga on vLLM, a state-of-the-art LLM inference engine, and evaluate it with diverse LLMs, datasets, and GPU configurations. Evaluations show that Jenga improves GPU memory utilization by up to 79.6%, and increases serving throughput by up to 4.92x (1.80x on average). |
| title | Jenga: Effective Memory Management for Serving LLM with Heterogeneity |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2503.18292 |