Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917364024475648 |
|---|---|
| author | Dietz, Laura Li, Bryan Yang, Eugene Lawrie, Dawn Walden, William Mayfield, James |
| author_facet | Dietz, Laura Li, Bryan Yang, Eugene Lawrie, Dawn Walden, William Mayfield, James |
| contents | RAG systems are increasingly evaluated and optimized using LLM judges, an approach that is rapidly becoming the dominant paradigm for system assessment. Nugget-based approaches in particular are now embedded not only in evaluation frameworks but also in the architectures of RAG systems themselves. While this integration can lead to genuine improvements, it also creates a risk of faulty measurements due to circularity. In this paper, we investigate this risk through comparative experiments with nugget-based RAG systems, including Ginger and Crucible, against strong baselines such as GPT-Researcher. By deliberately modifying Crucible to generate outputs optimized for an LLM judge, we show that near-perfect evaluation scores can be achieved when elements of the evaluation - such as prompt templates or gold nuggets - are leaked or can be predicted. Our results highlight the importance of blind evaluation settings and methodological diversity to guard against mistaking metric overfitting for genuine system progress. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_13227 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets? Dietz, Laura Li, Bryan Yang, Eugene Lawrie, Dawn Walden, William Mayfield, James Information Retrieval Artificial Intelligence H.3 RAG systems are increasingly evaluated and optimized using LLM judges, an approach that is rapidly becoming the dominant paradigm for system assessment. Nugget-based approaches in particular are now embedded not only in evaluation frameworks but also in the architectures of RAG systems themselves. While this integration can lead to genuine improvements, it also creates a risk of faulty measurements due to circularity. In this paper, we investigate this risk through comparative experiments with nugget-based RAG systems, including Ginger and Crucible, against strong baselines such as GPT-Researcher. By deliberately modifying Crucible to generate outputs optimized for an LLM judge, we show that near-perfect evaluation scores can be achieved when elements of the evaluation - such as prompt templates or gold nuggets - are leaked or can be predicted. Our results highlight the importance of blind evaluation settings and methodological diversity to guard against mistaking metric overfitting for genuine system progress. |
| title | Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets? |
| topic | Information Retrieval Artificial Intelligence H.3 |
| url | https://arxiv.org/abs/2601.13227 |