CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916525836861440 |
|---|---|
| author | Zhang, Hongxuan Zhao, Yao Zheng, Jiaqi Zhuang, Chenyi Gu, Jinjie Chen, Guihai |
| author_facet | Zhang, Hongxuan Zhao, Yao Zheng, Jiaqi Zhuang, Chenyi Gu, Jinjie Chen, Guihai |
| contents | The emergence of long-context text applications utilizing large language models (LLMs) has presented significant scalability challenges, particularly in memory footprint. The linear growth of the Key-Value (KV) cache responsible for storing attention keys and values to minimize redundant computations can lead to substantial increases in memory consumption, potentially causing models to fail to serve with limited memory resources. To address this issue, we propose a novel approach called Cache Sparse Representation (CSR), which converts the KV cache by transforming the dense Key-Value cache tensor into sparse indexes and weights, offering a more memory-efficient representation during LLM inference. Furthermore, we introduce NeuralDict, a novel neural network-based method for automatically generating the dictionary used in our sparse representation. Our extensive experiments demonstrate that CSR achieves performance comparable to state-of-the-art KV cache quantization algorithms while maintaining robust functionality in memory-constrained environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_11741 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation Zhang, Hongxuan Zhao, Yao Zheng, Jiaqi Zhuang, Chenyi Gu, Jinjie Chen, Guihai Computation and Language The emergence of long-context text applications utilizing large language models (LLMs) has presented significant scalability challenges, particularly in memory footprint. The linear growth of the Key-Value (KV) cache responsible for storing attention keys and values to minimize redundant computations can lead to substantial increases in memory consumption, potentially causing models to fail to serve with limited memory resources. To address this issue, we propose a novel approach called Cache Sparse Representation (CSR), which converts the KV cache by transforming the dense Key-Value cache tensor into sparse indexes and weights, offering a more memory-efficient representation during LLM inference. Furthermore, we introduce NeuralDict, a novel neural network-based method for automatically generating the dictionary used in our sparse representation. Our extensive experiments demonstrate that CSR achieves performance comparable to state-of-the-art KV cache quantization algorithms while maintaining robust functionality in memory-constrained environments. |
| title | CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.11741 |