FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918313160867840 |
|---|---|
| author | Kai, Jushi Wang, Yixuan Zeng, Boyi Bai, Haoli Jiang, Bo He, Ziwei Lin, Zhouhan |
| author_facet | Kai, Jushi Wang, Yixuan Zeng, Boyi Bai, Haoli Jiang, Bo He, Ziwei Lin, Zhouhan |
| contents | Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilling and decoding scenarios. When extrapolating beyond the pretrained context length, their performance degrades sharply on long-context benchmarks. Motivated by the observation in the frequency domain that the context information is concentrated in the low-frequency components, we propose FreqKV, a parameter-free and architecture-agnostic approach. It iteratively compresses the increasing KV cache in the frequency domain, allowing models to process lengthy contexts efficiently. With minimal training at 8K length, FreqKV extends the context window of LLaMA-2-7B up to 256K tokens while maintaining stable perplexity. Extensive experiments across prefilling and decoding demonstrate that FreqKV enables robust context window extension and consistently outperforms existing KV cache compression methods on LLaMA-2 and LLaMA-3, highlighting its effectiveness for both understanding and generation in long contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_00570 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension Kai, Jushi Wang, Yixuan Zeng, Boyi Bai, Haoli Jiang, Bo He, Ziwei Lin, Zhouhan Computation and Language Artificial Intelligence Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilling and decoding scenarios. When extrapolating beyond the pretrained context length, their performance degrades sharply on long-context benchmarks. Motivated by the observation in the frequency domain that the context information is concentrated in the low-frequency components, we propose FreqKV, a parameter-free and architecture-agnostic approach. It iteratively compresses the increasing KV cache in the frequency domain, allowing models to process lengthy contexts efficiently. With minimal training at 8K length, FreqKV extends the context window of LLaMA-2-7B up to 256K tokens while maintaining stable perplexity. Extensive experiments across prefilling and decoding demonstrate that FreqKV enables robust context window extension and consistently outperforms existing KV cache compression methods on LLaMA-2 and LLaMA-3, highlighting its effectiveness for both understanding and generation in long contexts. |
| title | FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.00570 |