EPIC: Efficient Position-Independent Caching for Serving Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912396541427712 |
|---|---|
| author | Hu, Junhao Huang, Wenrui Wang, Weidong Wang, Haoyi Hu, Tiancheng Zhang, Qin Feng, Hao Chen, Xusheng Shan, Yizhou Xie, Tao |
| author_facet | Hu, Junhao Huang, Wenrui Wang, Weidong Wang, Haoyi Hu, Tiancheng Zhang, Qin Feng, Hao Chen, Xusheng Shan, Yizhou Xie, Tao |
| contents | Large Language Models (LLMs) show great capabilities in a wide range of applications, but serving them efficiently becomes increasingly challenging as requests (prompts) become more complex. Context caching improves serving performance by reusing Key-Value (KV) vectors, the intermediate representations of tokens that are repeated across requests. However, existing context caching requires exact prefix matches across requests, limiting reuse cases in settings such as few-shot learning and retrieval-augmented generation, where immutable content (e.g., documents) remains unchanged across requests but is preceded by varying prefixes. Position-Independent Caching (PIC) addresses this issue by enabling modular reuse of the KV vectors regardless of prefixes. We formalize PIC and advance prior work by introducing EPIC, a serving system incorporating our new LegoLink algorithm, which mitigates the inappropriate "attention sink" effect at every document beginning, to maintain accuracy with minimal computation. Experiments show that EPIC achieves up to 8x improvements in Time-To-First-Token (TTFT) and 7x throughput gains over existing systems, with negligible or no accuracy loss. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15332 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EPIC: Efficient Position-Independent Caching for Serving Large Language Models Hu, Junhao Huang, Wenrui Wang, Weidong Wang, Haoyi Hu, Tiancheng Zhang, Qin Feng, Hao Chen, Xusheng Shan, Yizhou Xie, Tao Machine Learning Computation and Language Distributed, Parallel, and Cluster Computing Performance Large Language Models (LLMs) show great capabilities in a wide range of applications, but serving them efficiently becomes increasingly challenging as requests (prompts) become more complex. Context caching improves serving performance by reusing Key-Value (KV) vectors, the intermediate representations of tokens that are repeated across requests. However, existing context caching requires exact prefix matches across requests, limiting reuse cases in settings such as few-shot learning and retrieval-augmented generation, where immutable content (e.g., documents) remains unchanged across requests but is preceded by varying prefixes. Position-Independent Caching (PIC) addresses this issue by enabling modular reuse of the KV vectors regardless of prefixes. We formalize PIC and advance prior work by introducing EPIC, a serving system incorporating our new LegoLink algorithm, which mitigates the inappropriate "attention sink" effect at every document beginning, to maintain accuracy with minimal computation. Experiments show that EPIC achieves up to 8x improvements in Time-To-First-Token (TTFT) and 7x throughput gains over existing systems, with negligible or no accuracy loss. |
| title | EPIC: Efficient Position-Independent Caching for Serving Large Language Models |
| topic | Machine Learning Computation and Language Distributed, Parallel, and Cluster Computing Performance |
| url | https://arxiv.org/abs/2410.15332 |