FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908543150456832 |
|---|---|
| author | Wang, Dongwei Liu, Zijie Wang, Song Ren, Yuxin Deng, Jianing Hu, Jingtong Chen, Tianlong Yang, Huanrui |
| author_facet | Wang, Dongwei Liu, Zijie Wang, Song Ren, Yuxin Deng, Jianing Hu, Jingtong Chen, Tianlong Yang, Huanrui |
| contents | The Key-Value (KV) cache reading latency increases significantly with context lengths, hindering the efficiency of long-context LLM inference. To address this, previous works propose retaining a small fraction of KV cache based on token importance. For example, KV eviction uses static heuristics to retain tokens, while KV retrieval dynamically selects query-relevant tokens for more adaptive cache management. However, we observe that important tokens are often sparsely distributed across the long context. This sparsity makes existing page-level KV retrieval inaccurate, as each page may include irrelevant tokens and miss critical ones. In this work, we propose Fier, a \underline{Fi}ne-Grained and \underline{E}fficient KV cache \underline{R}etrieval method. Fier uses 1-bit quantized keys to estimate the importance of each token, resulting in efficient and precise retrieval. Experiments show that Fier matches full KV performance using only 11\% of the cache budget across various long-context tasks, reducing decoding latency by 1.2$\times$ to 1.5$\times$.Code is available at https://github.com/SimWangArizona/FIER |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_08256 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference Wang, Dongwei Liu, Zijie Wang, Song Ren, Yuxin Deng, Jianing Hu, Jingtong Chen, Tianlong Yang, Huanrui Databases The Key-Value (KV) cache reading latency increases significantly with context lengths, hindering the efficiency of long-context LLM inference. To address this, previous works propose retaining a small fraction of KV cache based on token importance. For example, KV eviction uses static heuristics to retain tokens, while KV retrieval dynamically selects query-relevant tokens for more adaptive cache management. However, we observe that important tokens are often sparsely distributed across the long context. This sparsity makes existing page-level KV retrieval inaccurate, as each page may include irrelevant tokens and miss critical ones. In this work, we propose Fier, a \underline{Fi}ne-Grained and \underline{E}fficient KV cache \underline{R}etrieval method. Fier uses 1-bit quantized keys to estimate the importance of each token, resulting in efficient and precise retrieval. Experiments show that Fier matches full KV performance using only 11\% of the cache budget across various long-context tasks, reducing decoding latency by 1.2$\times$ to 1.5$\times$.Code is available at https://github.com/SimWangArizona/FIER |
| title | FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference |
| topic | Databases |
| url | https://arxiv.org/abs/2508.08256 |