Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.20763 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908987268530176 |
|---|---|
| author | Klearman, Andrew Revutchi, Radu Garg, Rohin Chakravarti, Rishav Denton, Samuel Marc Xue, Yuan |
| author_facet | Klearman, Andrew Revutchi, Radu Garg, Rohin Chakravarti, Rishav Denton, Samuel Marc Xue, Yuan |
| contents | Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG). Current evaluation relies on heuristically constructed query sets, which introduce a hidden intrinsic bias. We formalize retrieval evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction. We further introduce \emph{semantic stratification}, which grounds evaluation in corpus structure by organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes.
Experiments across multiple benchmarks and retrieval methods validate our framework. The results expose systematic coverage gaps, identify structural signals that explain variance in retrieval performance, and show that stratified evaluation yields more stable and transparent assessments while supporting more trustworthy decision-making than aggregate metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_20763 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation Klearman, Andrew Revutchi, Radu Garg, Rohin Chakravarti, Rishav Denton, Samuel Marc Xue, Yuan Information Retrieval Artificial Intelligence Machine Learning Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG). Current evaluation relies on heuristically constructed query sets, which introduce a hidden intrinsic bias. We formalize retrieval evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction. We further introduce \emph{semantic stratification}, which grounds evaluation in corpus structure by organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes. Experiments across multiple benchmarks and retrieval methods validate our framework. The results expose systematic coverage gaps, identify structural signals that explain variance in retrieval performance, and show that stratified evaluation yields more stable and transparent assessments while supporting more trustworthy decision-making than aggregate metrics. |
| title | Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation |
| topic | Information Retrieval Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.20763 |