Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916634563706880 |
|---|---|
| author | Wang, Zilong Wang, Zifeng Le, Long Zheng, Huaixiu Steven Mishra, Swaroop Perot, Vincent Zhang, Yuwei Mattapalli, Anush Taly, Ankur Shang, Jingbo Lee, Chen-Yu Pfister, Tomas |
| author_facet | Wang, Zilong Wang, Zifeng Le, Long Zheng, Huaixiu Steven Mishra, Swaroop Perot, Vincent Zhang, Yuwei Mattapalli, Anush Taly, Ankur Shang, Jingbo Lee, Chen-Yu Pfister, Tomas |
| contents | Retrieval augmented generation (RAG) combines the generative abilities of large language models (LLMs) with external knowledge sources to provide more accurate and up-to-date responses. Recent RAG advancements focus on improving retrieval outcomes through iterative LLM refinement or self-critique capabilities acquired through additional instruction tuning of LLMs. In this work, we introduce Speculative RAG - a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Each draft is generated from a distinct subset of retrieved documents, offering diverse perspectives on the evidence while reducing input token counts per draft. This approach enhances comprehension of each subset and mitigates potential position bias over long context. Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts. Extensive experiments demonstrate that Speculative RAG achieves state-of-the-art performance with reduced latency on TriviaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge benchmarks. It notably enhances accuracy by up to 12.97% while reducing latency by 50.83% compared to conventional RAG systems on PubHealth. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_08223 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting Wang, Zilong Wang, Zifeng Le, Long Zheng, Huaixiu Steven Mishra, Swaroop Perot, Vincent Zhang, Yuwei Mattapalli, Anush Taly, Ankur Shang, Jingbo Lee, Chen-Yu Pfister, Tomas Computation and Language Artificial Intelligence Retrieval augmented generation (RAG) combines the generative abilities of large language models (LLMs) with external knowledge sources to provide more accurate and up-to-date responses. Recent RAG advancements focus on improving retrieval outcomes through iterative LLM refinement or self-critique capabilities acquired through additional instruction tuning of LLMs. In this work, we introduce Speculative RAG - a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Each draft is generated from a distinct subset of retrieved documents, offering diverse perspectives on the evidence while reducing input token counts per draft. This approach enhances comprehension of each subset and mitigates potential position bias over long context. Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts. Extensive experiments demonstrate that Speculative RAG achieves state-of-the-art performance with reduced latency on TriviaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge benchmarks. It notably enhances accuracy by up to 12.97% while reducing latency by 50.83% compared to conventional RAG systems on PubHealth. |
| title | Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.08223 |