Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.16161 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916658112626688 |
|---|---|
| author | Ispas, Alex-Razvan Simon, Charles-Elie Caspani, Fabien Guigue, Vincent |
| author_facet | Ispas, Alex-Razvan Simon, Charles-Elie Caspani, Fabien Guigue, Vincent |
| contents | Large Language Models are prompting us to view more NLP tasks from a generative perspective. At the same time, they offer a new way of accessing information, mainly through the RAG framework. While there have been notable improvements for the autoregressive models, overcoming hallucination in the generated answers remains a continuous problem. A standard solution is to use commercial LLMs, such as GPT4, to evaluate these algorithms. However, such frameworks are expensive and not very transparent. Therefore, we propose a study which demonstrates the interest of open-weight models for evaluating RAG hallucination. We develop a lightweight approach using smaller, quantized LLMs to provide an accessible and interpretable metric that gives continuous scores for the generated answer with respect to their correctness and faithfulness. This score allows us to question decisions' reliability and explore thresholds to develop a new AUC metric as an alternative to correlation with human judgment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_16161 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Lighter and Robust Evaluation for Retrieval Augmented Generation Ispas, Alex-Razvan Simon, Charles-Elie Caspani, Fabien Guigue, Vincent Computation and Language Artificial Intelligence 62-08 I.2.7 Large Language Models are prompting us to view more NLP tasks from a generative perspective. At the same time, they offer a new way of accessing information, mainly through the RAG framework. While there have been notable improvements for the autoregressive models, overcoming hallucination in the generated answers remains a continuous problem. A standard solution is to use commercial LLMs, such as GPT4, to evaluate these algorithms. However, such frameworks are expensive and not very transparent. Therefore, we propose a study which demonstrates the interest of open-weight models for evaluating RAG hallucination. We develop a lightweight approach using smaller, quantized LLMs to provide an accessible and interpretable metric that gives continuous scores for the generated answer with respect to their correctness and faithfulness. This score allows us to question decisions' reliability and explore thresholds to develop a new AUC metric as an alternative to correlation with human judgment. |
| title | Towards Lighter and Robust Evaluation for Retrieval Augmented Generation |
| topic | Computation and Language Artificial Intelligence 62-08 I.2.7 |
| url | https://arxiv.org/abs/2503.16161 |