Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.01519 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915768367579136 |
|---|---|
| author | Zhao, Shiju Hu, Junhao Zheng, Jiaqi Chen, Guihai |
| author_facet | Zhao, Shiju Hu, Junhao Zheng, Jiaqi Chen, Guihai |
| contents | The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Caching (PIC) has been proposed to enable KV reuse without positional constraints; however, existing approaches often incur substantial accuracy degradation, limiting their practical adoption. To address this issue, we propose native PIC by reintroducing the encoder to prevalent decoder-only LLMs and explicitly training it to support PIC. We further develop COMB, a PIC-aware caching system that integrates seamlessly with existing inference frameworks. Experimental results show that COMB reduces Time-to-First-Token (TTFT) by 51-94% and increases throughput by 3$\times$ with comparable accuracy. Furthermore, the quality improvement when using DeepSeek-V2-Lite-Chat demonstrates the applicability of COMB to other types of decoder-only LLMs. Our code is available at https://github.com/shijuzhao/Comb. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01519 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | You Need an Encoder for Native Position-Independent Caching Zhao, Shiju Hu, Junhao Zheng, Jiaqi Chen, Guihai Machine Learning Artificial Intelligence The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Caching (PIC) has been proposed to enable KV reuse without positional constraints; however, existing approaches often incur substantial accuracy degradation, limiting their practical adoption. To address this issue, we propose native PIC by reintroducing the encoder to prevalent decoder-only LLMs and explicitly training it to support PIC. We further develop COMB, a PIC-aware caching system that integrates seamlessly with existing inference frameworks. Experimental results show that COMB reduces Time-to-First-Token (TTFT) by 51-94% and increases throughput by 3$\times$ with comparable accuracy. Furthermore, the quality improvement when using DeepSeek-V2-Lite-Chat demonstrates the applicability of COMB to other types of decoder-only LLMs. Our code is available at https://github.com/shijuzhao/Comb. |
| title | You Need an Encoder for Native Position-Independent Caching |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.01519 |