BERGEN: A Benchmarking Library for Retrieval-Augmented Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929404705243136 |
|---|---|
| author | Rau, David Déjean, Hervé Chirkova, Nadezhda Formal, Thibault Wang, Shuai Nikoulina, Vassilina Clinchant, Stéphane |
| author_facet | Rau, David Déjean, Hervé Chirkova, Nadezhda Formal, Thibault Wang, Shuai Nikoulina, Vassilina Clinchant, Stéphane |
| contents | Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. In response to the recent popularity of generative LLMs, many RAG approaches have been proposed, which involve an intricate number of different configurations such as evaluation datasets, collections, metrics, retrievers, and LLMs. Inconsistent benchmarking poses a major challenge in comparing approaches and understanding the impact of each component in the pipeline. In this work, we study best practices that lay the groundwork for a systematic evaluation of RAG and present BERGEN, an end-to-end library for reproducible research standardizing RAG experiments. In an extensive study focusing on QA, we benchmark different state-of-the-art retrievers, rerankers, and LLMs. Additionally, we analyze existing RAG metrics and datasets. Our open-source library BERGEN is available under \url{https://github.com/naver/bergen}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01102 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BERGEN: A Benchmarking Library for Retrieval-Augmented Generation Rau, David Déjean, Hervé Chirkova, Nadezhda Formal, Thibault Wang, Shuai Nikoulina, Vassilina Clinchant, Stéphane Computation and Language Information Retrieval Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. In response to the recent popularity of generative LLMs, many RAG approaches have been proposed, which involve an intricate number of different configurations such as evaluation datasets, collections, metrics, retrievers, and LLMs. Inconsistent benchmarking poses a major challenge in comparing approaches and understanding the impact of each component in the pipeline. In this work, we study best practices that lay the groundwork for a systematic evaluation of RAG and present BERGEN, an end-to-end library for reproducible research standardizing RAG experiments. In an extensive study focusing on QA, we benchmark different state-of-the-art retrievers, rerankers, and LLMs. Additionally, we analyze existing RAG metrics and datasets. Our open-source library BERGEN is available under \url{https://github.com/naver/bergen}. |
| title | BERGEN: A Benchmarking Library for Retrieval-Augmented Generation |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2407.01102 |