xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917536927318016 |
|---|---|
| author | Chang, Chi-Chih Lin, Wei-Cheng Lin, Chien-Yu Chiang, Hung-Yueh Akhauri, Yash Dai, Xilai Jiang, Huiqiang Li, Yucheng Ceze, Luis Wu, Kai-Chiang Abdelfattah, Mohamed S. |
| author_facet | Chang, Chi-Chih Lin, Wei-Cheng Lin, Chien-Yu Chiang, Hung-Yueh Akhauri, Yash Dai, Xilai Jiang, Huiqiang Li, Yucheng Ceze, Luis Wu, Kai-Chiang Abdelfattah, Mohamed S. |
| contents | Long-context Large Language Models (LLMs) enable powerful applications but incur high memory costs due to the key-value states (KV-Cache). Recent studies attempt to share KV-Cache across layers, but these approaches either require expensive pretraining or rely on per-token cross-layer cosine similarity that is often limited in practice. We show, via Centered Kernel Alignment (CKA), that the dominant singular vectors of KV-Cache are well aligned across layers. Motivated by this observation, we propose xKV, a post-training compression method that jointly factorizes grouped-layer KV-Cache into a shared low-rank subspace, substantially reducing KV-Cache memory. Across widely used LLMs, xKV achieves up to 8x KV-Cache compression while preserving accuracy on long-context tasks and in multi-turn settings. To further improve efficiency, we introduce Selective Reconstruction (SR) at decode time. Combined with SR, xKV achieves up to 4.23x end-to-end speedup over the full attention baseline, and surpasses notable baselines with 30% higher throughput under a similar accuracy level. Overall, xKV provides a plug-and-play approach to reduce both memory and latency for long-context LLM inference. Our code is publicly available at: https://github.com/abdelfattah-lab/xKV. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_18893 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction Chang, Chi-Chih Lin, Wei-Cheng Lin, Chien-Yu Chiang, Hung-Yueh Akhauri, Yash Dai, Xilai Jiang, Huiqiang Li, Yucheng Ceze, Luis Wu, Kai-Chiang Abdelfattah, Mohamed S. Computation and Language Machine Learning Long-context Large Language Models (LLMs) enable powerful applications but incur high memory costs due to the key-value states (KV-Cache). Recent studies attempt to share KV-Cache across layers, but these approaches either require expensive pretraining or rely on per-token cross-layer cosine similarity that is often limited in practice. We show, via Centered Kernel Alignment (CKA), that the dominant singular vectors of KV-Cache are well aligned across layers. Motivated by this observation, we propose xKV, a post-training compression method that jointly factorizes grouped-layer KV-Cache into a shared low-rank subspace, substantially reducing KV-Cache memory. Across widely used LLMs, xKV achieves up to 8x KV-Cache compression while preserving accuracy on long-context tasks and in multi-turn settings. To further improve efficiency, we introduce Selective Reconstruction (SR) at decode time. Combined with SR, xKV achieves up to 4.23x end-to-end speedup over the full attention baseline, and surpasses notable baselines with 30% higher throughput under a similar accuracy level. Overall, xKV provides a plug-and-play approach to reduce both memory and latency for long-context LLM inference. Our code is publicly available at: https://github.com/abdelfattah-lab/xKV. |
| title | xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.18893 |