Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.05176 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912865413234688 |
|---|---|
| author | Zhang, Ji Li, Yiwei Feng, Shaoxiong Yuan, Peiwen Wang, Xinglin Shi, Jiayi Zhang, Yueqi Tan, Chuyi Pan, Boyuan Hu, Yao Li, Kan |
| author_facet | Zhang, Ji Li, Yiwei Feng, Shaoxiong Yuan, Peiwen Wang, Xinglin Shi, Jiayi Zhang, Yueqi Tan, Chuyi Pan, Boyuan Hu, Yao Li, Kan |
| contents | KV cache in autoregressive LLMs eliminates redundant recomputation but has emerged as the dominant memory and bandwidth bottleneck during inference, notably with long contexts and test-time scaling. KV quantization is a key lever for reducing cache cost, but accuracy drops sharply as the native KV distribution lacks flatness and thus maintains a wide quantization range. Prior work focuses on isolating outliers, which caps their error but fails to flatten the overall distribution, leaving performance fragile under low-bit settings. In this work, we show that the K cache maintains a stable, context-evolving structure, while the V cache carries latent semantic regularities, with both contributing to the organization of vectors into shared patterns. Building on these insights, we propose PatternKV, a pattern-aligned residual quantization scheme. It mines representative pattern vectors online, aligns each KV vector to its nearest pattern, and quantizes only the residual. This reshaping of the KV distribution flattens the quantization target and narrows its range, thereby improving the fidelity of low-bit KV quantization. Across long-context and test-time scaling settings on multiple backbones, PatternKV delivers consistent 2-bit gains, with a 0.08% average 4-bit drop relative to FP16, improves test-time scaling accuracy by 10% on average, and raises throughput by 1.5x while supporting 1.25x larger batches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_05176 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PatternKV: Flattening KV Representation Expands Quantization Headroom Zhang, Ji Li, Yiwei Feng, Shaoxiong Yuan, Peiwen Wang, Xinglin Shi, Jiayi Zhang, Yueqi Tan, Chuyi Pan, Boyuan Hu, Yao Li, Kan Machine Learning Artificial Intelligence KV cache in autoregressive LLMs eliminates redundant recomputation but has emerged as the dominant memory and bandwidth bottleneck during inference, notably with long contexts and test-time scaling. KV quantization is a key lever for reducing cache cost, but accuracy drops sharply as the native KV distribution lacks flatness and thus maintains a wide quantization range. Prior work focuses on isolating outliers, which caps their error but fails to flatten the overall distribution, leaving performance fragile under low-bit settings. In this work, we show that the K cache maintains a stable, context-evolving structure, while the V cache carries latent semantic regularities, with both contributing to the organization of vectors into shared patterns. Building on these insights, we propose PatternKV, a pattern-aligned residual quantization scheme. It mines representative pattern vectors online, aligns each KV vector to its nearest pattern, and quantizes only the residual. This reshaping of the KV distribution flattens the quantization target and narrows its range, thereby improving the fidelity of low-bit KV quantization. Across long-context and test-time scaling settings on multiple backbones, PatternKV delivers consistent 2-bit gains, with a 0.08% average 4-bit drop relative to FP16, improves test-time scaling accuracy by 10% on average, and raises throughput by 1.5x while supporting 1.25x larger batches. |
| title | PatternKV: Flattening KV Representation Expands Quantization Headroom |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.05176 |