Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915332910743552 |
|---|---|
| author | Guo, Yang Tao, Yutian Ming, Yifei Nowak, Robert D. Liang, Yingyu |
| author_facet | Guo, Yang Tao, Yutian Ming, Yifei Nowak, Robert D. Liang, Yingyu |
| contents | Retrieval-augmented generation (RAG) has seen many empirical successes in recent years by aiding the LLM with external knowledge. However, its theoretical aspect has remained mostly unexplored. In this paper, we propose the first finite-sample generalization bound for RAG in in-context linear regression and derive an exact bias-variance tradeoff. Our framework views the retrieved texts as query-dependent noisy in-context examples and recovers the classical in-context learning (ICL) and standard RAG as the limit cases. Our analysis suggests that an intrinsic ceiling on generalization error exists on RAG as opposed to the ICL. Furthermore, our framework is able to model retrieval both from the training data and from external corpora by introducing uniform and non-uniform RAG noise. In line with our theory, we show the sample efficiency of ICL and RAG empirically with experiments on common QA benchmarks, such as Natural Questions and TriviaQA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03100 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds Guo, Yang Tao, Yutian Ming, Yifei Nowak, Robert D. Liang, Yingyu Machine Learning Artificial Intelligence Computation and Language Information Retrieval Statistics Theory Retrieval-augmented generation (RAG) has seen many empirical successes in recent years by aiding the LLM with external knowledge. However, its theoretical aspect has remained mostly unexplored. In this paper, we propose the first finite-sample generalization bound for RAG in in-context linear regression and derive an exact bias-variance tradeoff. Our framework views the retrieved texts as query-dependent noisy in-context examples and recovers the classical in-context learning (ICL) and standard RAG as the limit cases. Our analysis suggests that an intrinsic ceiling on generalization error exists on RAG as opposed to the ICL. Furthermore, our framework is able to model retrieval both from the training data and from external corpora by introducing uniform and non-uniform RAG noise. In line with our theory, we show the sample efficiency of ICL and RAG empirically with experiments on common QA benchmarks, such as Natural Questions and TriviaQA. |
| title | Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds |
| topic | Machine Learning Artificial Intelligence Computation and Language Information Retrieval Statistics Theory |
| url | https://arxiv.org/abs/2506.03100 |