Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.20218 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911532629098496 |
|---|---|
| author | Cestola, Samuel Xia, Tianxiang Weiyan, Zheng Pengfei, Zheng Didona, Diego |
| author_facet | Cestola, Samuel Xia, Tianxiang Weiyan, Zheng Pengfei, Zheng Didona, Diego |
| contents | Retrieval-augmented generation improves large language models' accuracy by adding relevant retrieved text to the prompt. Chunk level caching (CLC) accelerates inference by precomputing KV caches for these retrieved chunks and reusing them. However, these caches miss cross-attention dependencies between chunks, which can reduce output quality. Several methods try to improve CLC accuracy using different techniques. We make two main contributions. First, we show that existing CLC approaches have fundamental limitations that limit their accuracy or their applicability. We back this conclusion with an extensive CLC system experimental evaluation. Second, we observe that existing CLC techniques are complementary. We leverage this insight to propose a new CLC design that carefully combines them and achieves better accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_20218 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | An experimental study of KV cache reuse strategies in chunk-level caching systems Cestola, Samuel Xia, Tianxiang Weiyan, Zheng Pengfei, Zheng Didona, Diego Computation and Language Machine Learning I.2.7 Retrieval-augmented generation improves large language models' accuracy by adding relevant retrieved text to the prompt. Chunk level caching (CLC) accelerates inference by precomputing KV caches for these retrieved chunks and reusing them. However, these caches miss cross-attention dependencies between chunks, which can reduce output quality. Several methods try to improve CLC accuracy using different techniques. We make two main contributions. First, we show that existing CLC approaches have fundamental limitations that limit their accuracy or their applicability. We back this conclusion with an extensive CLC system experimental evaluation. Second, we observe that existing CLC techniques are complementary. We leverage this insight to propose a new CLC design that carefully combines them and achieves better accuracy. |
| title | An experimental study of KV cache reuse strategies in chunk-level caching systems |
| topic | Computation and Language Machine Learning I.2.7 |
| url | https://arxiv.org/abs/2603.20218 |