SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911605218869248 |
|---|---|
| author | Tian, Jiayi Azizi, Seyedarmin Zhao, Yequan Potraghloo, Erfan Baghaei McPherson, Sean Sridhar, Sharath Nittur Wang, Zhengyang Zhang, Zheng Pedram, Massoud Kundu, Souvik |
| author_facet | Tian, Jiayi Azizi, Seyedarmin Zhao, Yequan Potraghloo, Erfan Baghaei McPherson, Sean Sridhar, Sharath Nittur Wang, Zhengyang Zhang, Zheng Pedram, Massoud Kundu, Souvik |
| contents | Large reasoning models (LRMs) often incur significant key-value (KV) cache overhead, due to their linear growth with the verbose chain-of-thought (CoT) reasoning. This incurs both memory overhead and throughput bottlenecks, limiting efficient deployment. To reduce KV cache size during inference, we first investigate the effectiveness of existing KV cache eviction methods for CoT reasoning. Interestingly, we find that due to unstable token-wise scoring and reduced effective KV budget caused by padding, state-of-the-art (SoTA) eviction methods fail to maintain accuracy in multi-batch settings. Additionally, these methods often generate longer sequences than the original model without eviction, as semantic-unaware token-wise eviction leads to repeated revalidation during reasoning. To address these issues, we present \textbf{SkipKV}, a \textbf{\textit{training-free}} KV compression method that performs selective \textit{eviction} and \textit{generation}, operating at a coarse-grained, sentence-level sequence removal for efficient CoT reasoning. In specific, it introduces a \textit{sentence-scoring metric} to identify and remove highly similar sentences while maintaining semantic coherence. To suppress redundant generation, SkipKV dynamically adjusts a steering vector to update the hidden activation states during inference, enforcing the LRM to generate concise responses. Extensive evaluations on multiple reasoning benchmarks demonstrate that SkipKV achieves up to $\mathbf{26.7}\%$ higher accuracy compared to baseline methods, at a similar compression budget. Additionally, compared to SoTA, SkipKV yields up to $\mathbf{1.6}\times$ shorter generation length while improving throughput by up to $\mathbf{1.7}\times$. Our code is released at: \href{https://github.com/TTTTTTris/SkipKV}{https://github.com/TTTTTTris/SkipKV}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_07993 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models Tian, Jiayi Azizi, Seyedarmin Zhao, Yequan Potraghloo, Erfan Baghaei McPherson, Sean Sridhar, Sharath Nittur Wang, Zhengyang Zhang, Zheng Pedram, Massoud Kundu, Souvik Artificial Intelligence Large reasoning models (LRMs) often incur significant key-value (KV) cache overhead, due to their linear growth with the verbose chain-of-thought (CoT) reasoning. This incurs both memory overhead and throughput bottlenecks, limiting efficient deployment. To reduce KV cache size during inference, we first investigate the effectiveness of existing KV cache eviction methods for CoT reasoning. Interestingly, we find that due to unstable token-wise scoring and reduced effective KV budget caused by padding, state-of-the-art (SoTA) eviction methods fail to maintain accuracy in multi-batch settings. Additionally, these methods often generate longer sequences than the original model without eviction, as semantic-unaware token-wise eviction leads to repeated revalidation during reasoning. To address these issues, we present \textbf{SkipKV}, a \textbf{\textit{training-free}} KV compression method that performs selective \textit{eviction} and \textit{generation}, operating at a coarse-grained, sentence-level sequence removal for efficient CoT reasoning. In specific, it introduces a \textit{sentence-scoring metric} to identify and remove highly similar sentences while maintaining semantic coherence. To suppress redundant generation, SkipKV dynamically adjusts a steering vector to update the hidden activation states during inference, enforcing the LRM to generate concise responses. Extensive evaluations on multiple reasoning benchmarks demonstrate that SkipKV achieves up to $\mathbf{26.7}\%$ higher accuracy compared to baseline methods, at a similar compression budget. Additionally, compared to SoTA, SkipKV yields up to $\mathbf{1.6}\times$ shorter generation length while improving throughput by up to $\mathbf{1.7}\times$. Our code is released at: \href{https://github.com/TTTTTTris/SkipKV}{https://github.com/TTTTTTris/SkipKV}. |
| title | SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2512.07993 |