Evaluating Generative Ad Hoc Information Retrieval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913358904557568 |
|---|---|
| author | Gienapp, Lukas Scells, Harrisen Deckers, Niklas Bevendorff, Janek Wang, Shuai Kiesel, Johannes Syed, Shahbaz Fröbe, Maik Zuccon, Guido Stein, Benno Hagen, Matthias Potthast, Martin |
| author_facet | Gienapp, Lukas Scells, Harrisen Deckers, Niklas Bevendorff, Janek Wang, Shuai Kiesel, Johannes Syed, Shahbaz Fröbe, Maik Zuccon, Guido Stein, Benno Hagen, Matthias Potthast, Martin |
| contents | Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_04694 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Evaluating Generative Ad Hoc Information Retrieval Gienapp, Lukas Scells, Harrisen Deckers, Niklas Bevendorff, Janek Wang, Shuai Kiesel, Johannes Syed, Shahbaz Fröbe, Maik Zuccon, Guido Stein, Benno Hagen, Matthias Potthast, Martin Information Retrieval Computation and Language Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization. |
| title | Evaluating Generative Ad Hoc Information Retrieval |
| topic | Information Retrieval Computation and Language |
| url | https://arxiv.org/abs/2311.04694 |