Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.23129 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910006267346944 |
|---|---|
| author | Hua, Yilun Castellucci, Giuseppe Schulam, Peter Elfardy, Heba Small, Kevin |
| author_facet | Hua, Yilun Castellucci, Giuseppe Schulam, Peter Elfardy, Heba Small, Kevin |
| contents | Retrieval Augmented Generation (RAG)'s success depends on the utility the LLM derives from the content used for grounding. Quantifying content utility does not have a definitive specification and existing metrics ignore model-specific capabilities and/or rely on costly annotations. In this paper, we propose Grounding Generation Utility (GroGU), a model-specific and reference-free metric that defines utility as a function of the downstream LLM's generation confidence based on entropy. Despite having no annotation requirements, GroGU is largely faithful in distinguishing ground-truth documents while capturing nuances ignored by LLM-agnostic metrics. We apply GroGU to train a query-rewriter for RAG by identifying high-utility preference data for Direct Preference Optimization. Experiments show improvements by up to 18.2 points in Mean Reciprocal Rank and up to 9.4 points in answer accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_23129 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluating the Utility of Grounding Documents with Reference-Free LLM-based Metrics Hua, Yilun Castellucci, Giuseppe Schulam, Peter Elfardy, Heba Small, Kevin Computation and Language Retrieval Augmented Generation (RAG)'s success depends on the utility the LLM derives from the content used for grounding. Quantifying content utility does not have a definitive specification and existing metrics ignore model-specific capabilities and/or rely on costly annotations. In this paper, we propose Grounding Generation Utility (GroGU), a model-specific and reference-free metric that defines utility as a function of the downstream LLM's generation confidence based on entropy. Despite having no annotation requirements, GroGU is largely faithful in distinguishing ground-truth documents while capturing nuances ignored by LLM-agnostic metrics. We apply GroGU to train a query-rewriter for RAG by identifying high-utility preference data for Direct Preference Optimization. Experiments show improvements by up to 18.2 points in Mean Reciprocal Rank and up to 9.4 points in answer accuracy. |
| title | Evaluating the Utility of Grounding Documents with Reference-Free LLM-based Metrics |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.23129 |