Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909959421165568 |
|---|---|
| author | Liu, Minghui Palnitkar, Aadi Rabbani, Tahseen Jae, Hyunwoo Sang, Kyle Rui Yao, Dixi Shabihi, Shayan Zhao, Fuheng Li, Tian Zhang, Ce Huang, Furong Zhang, Kunpeng |
| author_facet | Liu, Minghui Palnitkar, Aadi Rabbani, Tahseen Jae, Hyunwoo Sang, Kyle Rui Yao, Dixi Shabihi, Shayan Zhao, Fuheng Li, Tian Zhang, Ce Huang, Furong Zhang, Kunpeng |
| contents | Large language models (LLMs) have demonstrated remarkable performance on long-context tasks, but are often bottlenecked by memory constraints. Namely, the KV cache, which is used to significantly speed up attention computations, grows linearly with context length. A suite of compression algorithms has been introduced to alleviate cache growth by evicting unimportant tokens. However, several popular strategies are targeted towards the prefill phase, i.e., processing long prompt context, and their performance is rarely assessed on reasoning tasks requiring long decoding. In particular, short but complex prompts, such as those in benchmarks like GSM8K and MATH500, often benefit from multi-step reasoning and self-reflection, resulting in thinking sequences thousands of tokens long. In this work, we benchmark the performance of several popular compression strategies on long-reasoning tasks. For the non-reasoning Llama-3.1-8B-Instruct, we determine that no singular strategy fits all, and that performance is heavily influenced by dataset type. However, we discover that H2O and our decoding-enabled variant of SnapKV are dominant strategies for reasoning models, indicating the utility of heavy-hitter tracking for reasoning traces. We also find that eviction strategies at low budgets can produce longer reasoning traces, revealing a tradeoff between cache size and inference costs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_12008 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hold Onto That Thought: Assessing KV Cache Compression On Reasoning Liu, Minghui Palnitkar, Aadi Rabbani, Tahseen Jae, Hyunwoo Sang, Kyle Rui Yao, Dixi Shabihi, Shayan Zhao, Fuheng Li, Tian Zhang, Ce Huang, Furong Zhang, Kunpeng Computation and Language Artificial Intelligence Performance Large language models (LLMs) have demonstrated remarkable performance on long-context tasks, but are often bottlenecked by memory constraints. Namely, the KV cache, which is used to significantly speed up attention computations, grows linearly with context length. A suite of compression algorithms has been introduced to alleviate cache growth by evicting unimportant tokens. However, several popular strategies are targeted towards the prefill phase, i.e., processing long prompt context, and their performance is rarely assessed on reasoning tasks requiring long decoding. In particular, short but complex prompts, such as those in benchmarks like GSM8K and MATH500, often benefit from multi-step reasoning and self-reflection, resulting in thinking sequences thousands of tokens long. In this work, we benchmark the performance of several popular compression strategies on long-reasoning tasks. For the non-reasoning Llama-3.1-8B-Instruct, we determine that no singular strategy fits all, and that performance is heavily influenced by dataset type. However, we discover that H2O and our decoding-enabled variant of SnapKV are dominant strategies for reasoning models, indicating the utility of heavy-hitter tracking for reasoning traces. We also find that eviction strategies at low budgets can produce longer reasoning traces, revealing a tradeoff between cache size and inference costs. |
| title | Hold Onto That Thought: Assessing KV Cache Compression On Reasoning |
| topic | Computation and Language Artificial Intelligence Performance |
| url | https://arxiv.org/abs/2512.12008 |