TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916023210344448 |
|---|---|
| author | Lin, Chien-Yu Kamahori, Keisuke Liu, Yiyu Shi, Xiaoxiang Kashyap, Madhav Gu, Yile Shao, Rulin Ye, Zihao Zhu, Kan Kadekodi, Rohan Wang, Stephanie Krishnamurthy, Arvind Ceze, Luis Kasikci, Baris |
| author_facet | Lin, Chien-Yu Kamahori, Keisuke Liu, Yiyu Shi, Xiaoxiang Kashyap, Madhav Gu, Yile Shao, Rulin Ye, Zihao Zhu, Kan Kadekodi, Rohan Wang, Stephanie Krishnamurthy, Arvind Ceze, Luis Kasikci, Baris |
| contents | Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and domain coverage. Modern RAG pipelines rely on large datastores, creating a significant system challenge: achieving high throughput and low latency is difficult, especially when GPU memory is limited. To address these challenges, we propose TeleRAG, an efficient inference system that reduces latency and improves throughput with minimal GPU memory requirements. The core innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that predicts required data and transfers them from CPU to GPU in parallel with LLM generation. In addition, TeleRAG adopts a prefetching scheduler and a cache-aware scheduler to support efficient multi-GPU inference with minimal overhead. Evaluations show TeleRAG achieves up to a 1.53x average end-to-end latency reduction (single-query) and 1.83x higher average throughput (batched), as well as good scalability in throughput. This confirms the practical utility of TeleRAG for faster and more memory-efficient deployments of RAG applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20969 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval Lin, Chien-Yu Kamahori, Keisuke Liu, Yiyu Shi, Xiaoxiang Kashyap, Madhav Gu, Yile Shao, Rulin Ye, Zihao Zhu, Kan Kadekodi, Rohan Wang, Stephanie Krishnamurthy, Arvind Ceze, Luis Kasikci, Baris Distributed, Parallel, and Cluster Computing Machine Learning Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and domain coverage. Modern RAG pipelines rely on large datastores, creating a significant system challenge: achieving high throughput and low latency is difficult, especially when GPU memory is limited. To address these challenges, we propose TeleRAG, an efficient inference system that reduces latency and improves throughput with minimal GPU memory requirements. The core innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that predicts required data and transfers them from CPU to GPU in parallel with LLM generation. In addition, TeleRAG adopts a prefetching scheduler and a cache-aware scheduler to support efficient multi-GPU inference with minimal overhead. Evaluations show TeleRAG achieves up to a 1.53x average end-to-end latency reduction (single-query) and 1.83x higher average throughput (batched), as well as good scalability in throughput. This confirms the practical utility of TeleRAG for faster and more memory-efficient deployments of RAG applications. |
| title | TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2502.20969 |