Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911500515409920 |
|---|---|
| author | Zheng, Xianzhe Wang, Zhengheng Ma, Ruiyan Wang, Rui Wang, Xiyu Chen, Rui Zhang, Peng Pan, Sicheng Huang, Zhangheng Wu, Chenxin Zhang, Yi Cai, Bo Liu, Kan Ma, Teng Du, Yin Deng, Dong Wu, Sai Zhu, Guoyun Zhang, Wei Li, Feifei |
| author_facet | Zheng, Xianzhe Wang, Zhengheng Ma, Ruiyan Wang, Rui Wang, Xiyu Chen, Rui Zhang, Peng Pan, Sicheng Huang, Zhangheng Wu, Chenxin Zhang, Yi Cai, Bo Liu, Kan Ma, Teng Du, Yin Deng, Dong Wu, Sai Zhu, Guoyun Zhang, Wei Li, Feifei |
| contents | The memory-for-computation paradigm of KV caching is essential for accelerating large language model (LLM) inference service, but limited GPU high-bandwidth memory (HBM) capacity motivates offloading the KV cache to cheaper external storage tiers. While this expands capacity, it introduces the challenge of dynamically managing heterogeneous storage resources to balance cost, throughput, and latency under varying workloads. We formulate this as a multi-objective optimization problem: identifying the Pareto frontier across these metrics within the storage configuration space. Using a high-fidelity end-to-end simulator, we observe that the objective functions are non-analytic and exhibit complex variable coupling, making the Pareto frontier difficult to approximate analytically. To obtain the frontier, we introduce Kareto, a KV-cache Adaptive REsource managemenT Optimizer. Kareto leverages a diminishing-return-guided pruning method to efficiently navigate the large configuration space and approximate the Pareto frontier. Additionally, it incorporates a fine-grained adaptive tuner that uses eviction policies in tier storage and KV block access patterns for group-specific cache management, improving cache efficiency. Experiments on real-world traces show that Kareto adapts to workload and can identify configurations of better cost efficiency, covering static strategies. Compared to the fixed setup with 1024 GB DRAM, Kareto can improve throughput by up to 9.3%, or reduce latency by up to 58.3%, or lower cost by up to 20.2% under respective optimization objectives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_08739 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service Zheng, Xianzhe Wang, Zhengheng Ma, Ruiyan Wang, Rui Wang, Xiyu Chen, Rui Zhang, Peng Pan, Sicheng Huang, Zhangheng Wu, Chenxin Zhang, Yi Cai, Bo Liu, Kan Ma, Teng Du, Yin Deng, Dong Wu, Sai Zhu, Guoyun Zhang, Wei Li, Feifei Hardware Architecture Distributed, Parallel, and Cluster Computing The memory-for-computation paradigm of KV caching is essential for accelerating large language model (LLM) inference service, but limited GPU high-bandwidth memory (HBM) capacity motivates offloading the KV cache to cheaper external storage tiers. While this expands capacity, it introduces the challenge of dynamically managing heterogeneous storage resources to balance cost, throughput, and latency under varying workloads. We formulate this as a multi-objective optimization problem: identifying the Pareto frontier across these metrics within the storage configuration space. Using a high-fidelity end-to-end simulator, we observe that the objective functions are non-analytic and exhibit complex variable coupling, making the Pareto frontier difficult to approximate analytically. To obtain the frontier, we introduce Kareto, a KV-cache Adaptive REsource managemenT Optimizer. Kareto leverages a diminishing-return-guided pruning method to efficiently navigate the large configuration space and approximate the Pareto frontier. Additionally, it incorporates a fine-grained adaptive tuner that uses eviction policies in tier storage and KV block access patterns for group-specific cache management, improving cache efficiency. Experiments on real-world traces show that Kareto adapts to workload and can identify configurations of better cost efficiency, covering static strategies. Compared to the fixed setup with 1024 GB DRAM, Kareto can improve throughput by up to 9.3%, or reduce latency by up to 58.3%, or lower cost by up to 20.2% under respective optimization objectives. |
| title | Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service |
| topic | Hardware Architecture Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2603.08739 |