Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.16048 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914971106934784 |
|---|---|
| author | Rassin, Royi Fairstein, Yaron Kalinsky, Oren Kushilevitz, Guy Cohen, Nachshon Libov, Alexander Goldberg, Yoav |
| author_facet | Rassin, Royi Fairstein, Yaron Kalinsky, Oren Kushilevitz, Guy Cohen, Nachshon Libov, Alexander Goldberg, Yoav |
| contents | Retrieval models are often evaluated on partially-annotated datasets. Each query is mapped to a few relevant texts and the remaining corpus is assumed to be irrelevant. As a result, models that successfully retrieve false negatives are punished in evaluation. Unfortunately, completely annotating all texts for every query is not resource efficient. In this work, we show that using partially-annotated datasets in evaluation can paint a distorted picture. We curate D-MERIT, a passage retrieval evaluation set from Wikipedia, aspiring to contain all relevant passages for each query. Queries describe a group (e.g., "journals about linguistics") and relevant passages are evidence that entities belong to the group (e.g., a passage indicating that "Language" is a journal about linguistics). We show that evaluating on a dataset containing annotations for only a subset of the relevant passages might result in misleading ranking of the retrieval systems and that as more relevant texts are included in the evaluation set, the rankings converge. We propose our dataset as a resource for evaluation and our study as a recommendation for balance between resource-efficiency and reliable evaluation when annotating evaluation sets for text retrieval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16048 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating D-MERIT of Partial-annotation on Information Retrieval Rassin, Royi Fairstein, Yaron Kalinsky, Oren Kushilevitz, Guy Cohen, Nachshon Libov, Alexander Goldberg, Yoav Information Retrieval Retrieval models are often evaluated on partially-annotated datasets. Each query is mapped to a few relevant texts and the remaining corpus is assumed to be irrelevant. As a result, models that successfully retrieve false negatives are punished in evaluation. Unfortunately, completely annotating all texts for every query is not resource efficient. In this work, we show that using partially-annotated datasets in evaluation can paint a distorted picture. We curate D-MERIT, a passage retrieval evaluation set from Wikipedia, aspiring to contain all relevant passages for each query. Queries describe a group (e.g., "journals about linguistics") and relevant passages are evidence that entities belong to the group (e.g., a passage indicating that "Language" is a journal about linguistics). We show that evaluating on a dataset containing annotations for only a subset of the relevant passages might result in misleading ranking of the retrieval systems and that as more relevant texts are included in the evaluation set, the rankings converge. We propose our dataset as a resource for evaluation and our study as a recommendation for balance between resource-efficiency and reliable evaluation when annotating evaluation sets for text retrieval. |
| title | Evaluating D-MERIT of Partial-annotation on Information Retrieval |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2406.16048 |