Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912462748516352 |
|---|---|
| author | Metel, Michael R. Chen, Boxing Rezagholizadeh, Mehdi |
| author_facet | Metel, Michael R. Chen, Boxing Rezagholizadeh, Mehdi |
| contents | Several works have developed eviction policies to remove key-value (KV) pairs from the KV cache for more efficient inference. The focus has been on compressing the KV cache after the input prompt has been processed for faster token generation. In settings with limited GPU memory, and when the input context is longer than the generation length, we show that by also compressing the KV cache during the input processing phase, larger batch sizes can be used resulting in significantly higher throughput while still maintaining the original model's accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_05693 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression Metel, Michael R. Chen, Boxing Rezagholizadeh, Mehdi Computation and Language Several works have developed eviction policies to remove key-value (KV) pairs from the KV cache for more efficient inference. The focus has been on compressing the KV cache after the input prompt has been processed for faster token generation. In settings with limited GPU memory, and when the input context is longer than the generation length, we show that by also compressing the KV cache during the input processing phase, larger batch sizes can be used resulting in significantly higher throughput while still maintaining the original model's accuracy. |
| title | Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.05693 |