XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911207266451456 |
|---|---|
| author | Yang, Haoqi Yao, Yao Li, Zuchao Qi, Baoyuan Liu, Guoming Zhao, Hai |
| author_facet | Yang, Haoqi Yao, Yao Li, Zuchao Qi, Baoyuan Liu, Guoming Zhao, Hai |
| contents | Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks. However, their extensive memory requirements, particularly due to KV cache growth during long-text understanding and generation, present significant challenges for deployment in resource-constrained environments. Quantization has emerged as a promising solution to reduce memory consumption while preserving historical information. We propose XQuant, a training-free and plug-and-play framework that achieves ultra-low equivalent bit-width KV cache quantization. XQuant introduces two key innovations: a computationally negligible data-free calibration method and cross-layer KV cache compression, enabling quantization to sub-1.4 bits. Extensive experiments on TruthfulQA and LongBench demonstrate that XQuant outperforms state-of-the-art methods (e.g., KIVI-2bit and AsymKV-1.5bit) by achieving lower bit-width while maintaining superior performance, establishing a better trade-off between memory efficiency and model accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_11236 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression Yang, Haoqi Yao, Yao Li, Zuchao Qi, Baoyuan Liu, Guoming Zhao, Hai Computation and Language Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks. However, their extensive memory requirements, particularly due to KV cache growth during long-text understanding and generation, present significant challenges for deployment in resource-constrained environments. Quantization has emerged as a promising solution to reduce memory consumption while preserving historical information. We propose XQuant, a training-free and plug-and-play framework that achieves ultra-low equivalent bit-width KV cache quantization. XQuant introduces two key innovations: a computationally negligible data-free calibration method and cross-layer KV cache compression, enabling quantization to sub-1.4 bits. Extensive experiments on TruthfulQA and LongBench demonstrate that XQuant outperforms state-of-the-art methods (e.g., KIVI-2bit and AsymKV-1.5bit) by achieving lower bit-width while maintaining superior performance, establishing a better trade-off between memory efficiency and model accuracy. |
| title | XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.11236 |