HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modeling
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910221795852288 |
|---|---|
| author | Cederlund, Jonathan Berg, Axel Acar, Durmus Alp Emre Zhou, Chuteng Giselsson, Pontus |
| author_facet | Cederlund, Jonathan Berg, Axel Acar, Durmus Alp Emre Zhou, Chuteng Giselsson, Pontus |
| contents | Visual Autoregressive (VAR) models have recently demonstrated impressive image generation quality while maintaining low latency. However, they suffer from severe KV-cache memory constraints, often requiring gigabytes of memory per generated image. We introduce HeatKV, a novel compression method that adapts cache allocation in each head based on its attention to previously generated scales. Using a small offline calibration set, the attention heads are ranked according to their attention scores over prior scales. Based on this ranking, we construct a static pruning schedule tailored to a given memory budget. Applied to the Infinity-2B model, HeatKV achieves $2 \times$ higher compression ratio in memory allocation for KV cache compared to existing methods, while maintaining similar or better image fidelity, prompt alignment and human perception score. Our method achieves a new state-of-the-art (SOTA) for VAR model KV-cache compression, showcasing the effectiveness of fine-grained, head-specific cache allocation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_14877 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modeling Cederlund, Jonathan Berg, Axel Acar, Durmus Alp Emre Zhou, Chuteng Giselsson, Pontus Computer Vision and Pattern Recognition Visual Autoregressive (VAR) models have recently demonstrated impressive image generation quality while maintaining low latency. However, they suffer from severe KV-cache memory constraints, often requiring gigabytes of memory per generated image. We introduce HeatKV, a novel compression method that adapts cache allocation in each head based on its attention to previously generated scales. Using a small offline calibration set, the attention heads are ranked according to their attention scores over prior scales. Based on this ranking, we construct a static pruning schedule tailored to a given memory budget. Applied to the Infinity-2B model, HeatKV achieves $2 \times$ higher compression ratio in memory allocation for KV cache compared to existing methods, while maintaining similar or better image fidelity, prompt alignment and human perception score. Our method achieves a new state-of-the-art (SOTA) for VAR model KV-cache compression, showcasing the effectiveness of fine-grained, head-specific cache allocation. |
| title | HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modeling |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.14877 |