Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.03759 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914939642314752 |
|---|---|
| author | Ding, Tianyu Banerjee, Adi Mombaerts, Laurent Li, Yunhong Borogovac, Tarik Weinstein, Juan Pablo De la Cruz |
| author_facet | Ding, Tianyu Banerjee, Adi Mombaerts, Laurent Li, Yunhong Borogovac, Tarik Weinstein, Juan Pablo De la Cruz |
| contents | The increasing use of Retrieval-Augmented Generation (RAG) systems in various applications necessitates stringent protocols to ensure RAG systems accuracy, safety, and alignment with user intentions. In this paper, we introduce VERA (Validation and Evaluation of Retrieval-Augmented Systems), a framework designed to enhance the transparency and reliability of outputs from large language models (LLMs) that utilize retrieved information. VERA improves the way we evaluate RAG systems in two important ways: (1) it introduces a cross-encoder based mechanism that encompasses a set of multidimensional metrics into a single comprehensive ranking score, addressing the challenge of prioritizing individual metrics, and (2) it employs Bootstrap statistics on LLM-based metrics across the document repository to establish confidence bounds, ensuring the repositorys topical coverage and improving the overall reliability of retrieval systems. Through several use cases, we demonstrate how VERA can strengthen decision-making processes and trust in AI applications. Our findings not only contribute to the theoretical understanding of LLM-based RAG evaluation metric but also promote the practical implementation of responsible AI systems, marking a significant advancement in the development of reliable and transparent generative AI technologies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_03759 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VERA: Validation and Evaluation of Retrieval-Augmented Systems Ding, Tianyu Banerjee, Adi Mombaerts, Laurent Li, Yunhong Borogovac, Tarik Weinstein, Juan Pablo De la Cruz Information Retrieval Artificial Intelligence I.2.7 The increasing use of Retrieval-Augmented Generation (RAG) systems in various applications necessitates stringent protocols to ensure RAG systems accuracy, safety, and alignment with user intentions. In this paper, we introduce VERA (Validation and Evaluation of Retrieval-Augmented Systems), a framework designed to enhance the transparency and reliability of outputs from large language models (LLMs) that utilize retrieved information. VERA improves the way we evaluate RAG systems in two important ways: (1) it introduces a cross-encoder based mechanism that encompasses a set of multidimensional metrics into a single comprehensive ranking score, addressing the challenge of prioritizing individual metrics, and (2) it employs Bootstrap statistics on LLM-based metrics across the document repository to establish confidence bounds, ensuring the repositorys topical coverage and improving the overall reliability of retrieval systems. Through several use cases, we demonstrate how VERA can strengthen decision-making processes and trust in AI applications. Our findings not only contribute to the theoretical understanding of LLM-based RAG evaluation metric but also promote the practical implementation of responsible AI systems, marking a significant advancement in the development of reliable and transparent generative AI technologies. |
| title | VERA: Validation and Evaluation of Retrieval-Augmented Systems |
| topic | Information Retrieval Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2409.03759 |