Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912912579231744 |
|---|---|
| author | Lin, Hongzhan Bai, Zhiqi Zhang, Xinmiao Yang, Sen Li, Xiang Yang, Siran Xu, Yunlong Liu, Jiaheng Zhao, Yongchi Wang, Jiamang Xu, Yuchi Su, Wenbo Zheng, Bo |
| author_facet | Lin, Hongzhan Bai, Zhiqi Zhang, Xinmiao Yang, Sen Li, Xiang Yang, Siran Xu, Yunlong Liu, Jiaheng Zhao, Yongchi Wang, Jiamang Xu, Yuchi Su, Wenbo Zheng, Bo |
| contents | Transformer decoders have achieved strong results across tasks, but the memory required for the KV cache becomes prohibitive at long sequence lengths. Although Cross-layer KV Cache sharing (e.g., YOCO, CLA) offers a path to mitigate KV Cache bottleneck, it typically underperforms within-layer methods like GQA. To understand the root cause, we investigate the information flow of keys and values of the top-layers. Our preliminary reveals a clear distribution: values are predominantly derived from the bottom layer, while keys draw more information from both bottom and middle layers. Building upon this, we propose FusedKV, whose top-layer KV caches are a learnable fusion of the most informative ones from the bottom and middle layers. This fusion operates directly on post-RoPE keys, preserving relative positional information without the computational cost of re-applying rotary embeddings. To further improve efficiency, we propose FusedKV-Lite, an cross-layer sharing approach, where top-layer KV caches are directly derived from the bottom-layer values and the middle-layer keys. Compared to FusedKV, FusedKV-Lite reduces I/O overhead at the cost of a slight increase in perplexity. In experiments on LLMs ranging from 332M to 4B parameters, our proposed method reduce 50\% cache memory while achieving lower validation perplexity than the standard Transformer decoder, establishing it as a memory-efficient, high-performance architectural alternative. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_03870 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers Lin, Hongzhan Bai, Zhiqi Zhang, Xinmiao Yang, Sen Li, Xiang Yang, Siran Xu, Yunlong Liu, Jiaheng Zhao, Yongchi Wang, Jiamang Xu, Yuchi Su, Wenbo Zheng, Bo Computation and Language Transformer decoders have achieved strong results across tasks, but the memory required for the KV cache becomes prohibitive at long sequence lengths. Although Cross-layer KV Cache sharing (e.g., YOCO, CLA) offers a path to mitigate KV Cache bottleneck, it typically underperforms within-layer methods like GQA. To understand the root cause, we investigate the information flow of keys and values of the top-layers. Our preliminary reveals a clear distribution: values are predominantly derived from the bottom layer, while keys draw more information from both bottom and middle layers. Building upon this, we propose FusedKV, whose top-layer KV caches are a learnable fusion of the most informative ones from the bottom and middle layers. This fusion operates directly on post-RoPE keys, preserving relative positional information without the computational cost of re-applying rotary embeddings. To further improve efficiency, we propose FusedKV-Lite, an cross-layer sharing approach, where top-layer KV caches are directly derived from the bottom-layer values and the middle-layer keys. Compared to FusedKV, FusedKV-Lite reduces I/O overhead at the cost of a slight increase in perplexity. In experiments on LLMs ranging from 332M to 4B parameters, our proposed method reduce 50\% cache memory while achieving lower validation perplexity than the standard Transformer decoder, establishing it as a memory-efficient, high-performance architectural alternative. |
| title | Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.03870 |