A Method for Building Large Language Models with Predefined KV Cache Capacity
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910717587750912 |
|---|---|
| author | Yi, Zhonghua Niu, Ge Wang, Lei Tang, Wei Zhang, Liqiu |
| author_facet | Yi, Zhonghua Niu, Ge Wang, Lei Tang, Wei Zhang, Liqiu |
| contents | This paper introduces a novel approach, the Bounded-Cache Transformer (BCT), for building large language models with a predefined Key-Value (KV) cache capacity. The BCT addresses the excessive memory consumption issue in traditional KV caches by implementing a bounded-length KV cache, which is particularly suitable for the attention layers in Transformer decode-only architectures. By dynamically updating the key-value vector sequences, the BCT achieves efficient inference within limited cache capacity, significantly reducing memory usage while maintaining model performance and system throughput. Experimental results demonstrate that the BCT significantly reduces memory usage while maintaining the model's inference quality, offering a new solution for efficient inference in large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_15785 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Method for Building Large Language Models with Predefined KV Cache Capacity Yi, Zhonghua Niu, Ge Wang, Lei Tang, Wei Zhang, Liqiu Computation and Language This paper introduces a novel approach, the Bounded-Cache Transformer (BCT), for building large language models with a predefined Key-Value (KV) cache capacity. The BCT addresses the excessive memory consumption issue in traditional KV caches by implementing a bounded-length KV cache, which is particularly suitable for the attention layers in Transformer decode-only architectures. By dynamically updating the key-value vector sequences, the BCT achieves efficient inference within limited cache capacity, significantly reducing memory usage while maintaining model performance and system throughput. Experimental results demonstrate that the BCT significantly reduces memory usage while maintaining the model's inference quality, offering a new solution for efficient inference in large language models. |
| title | A Method for Building Large Language Models with Predefined KV Cache Capacity |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.15785 |