Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918141013000192 |
|---|---|
| author | Fang, Yunhua Xie, Rui Haq, Asad Ul Ma, Linsen Maghraoui, Kaoutar El Wang, Naigang Wang, Meng Liu, Liu Zhang, Tong |
| author_facet | Fang, Yunhua Xie, Rui Haq, Asad Ul Ma, Linsen Maghraoui, Kaoutar El Wang, Naigang Wang, Meng Liu, Liu Zhang, Tong |
| contents | Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity reduces some memory traffic, the relevance of past tokens varies over time, requiring the full KV cache to remain accessible and sustaining pressure on both bandwidth and capacity. With advances in interconnects such as NVLink and LPDDR5X, modern AI hardware now integrates high-bandwidth memory (HBM) with high-speed off-package DRAM, making heterogeneous memory systems a practical solution. This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization. To our knowledge, this is the first formal treatment of dynamic KV cache scheduling in heterogeneous memory systems for LLM inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_13231 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System Fang, Yunhua Xie, Rui Haq, Asad Ul Ma, Linsen Maghraoui, Kaoutar El Wang, Naigang Wang, Meng Liu, Liu Zhang, Tong Hardware Architecture Artificial Intelligence Performance Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity reduces some memory traffic, the relevance of past tokens varies over time, requiring the full KV cache to remain accessible and sustaining pressure on both bandwidth and capacity. With advances in interconnects such as NVLink and LPDDR5X, modern AI hardware now integrates high-bandwidth memory (HBM) with high-speed off-package DRAM, making heterogeneous memory systems a practical solution. This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization. To our knowledge, this is the first formal treatment of dynamic KV cache scheduling in heterogeneous memory systems for LLM inference. |
| title | Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System |
| topic | Hardware Architecture Artificial Intelligence Performance |
| url | https://arxiv.org/abs/2508.13231 |