Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.07584 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916683387502592 |
|---|---|
| author | Engelmann, Björn Haak, Fabian Schaer, Philipp Abdoust, Mani Erfanian Netze, Linus Bittkowski, Meik |
| author_facet | Engelmann, Björn Haak, Fabian Schaer, Philipp Abdoust, Mani Erfanian Netze, Linus Bittkowski, Meik |
| contents | Retrieval test collections are essential for evaluating information retrieval systems, yet they often lack generalizability across tasks. To overcome this limitation, we introduce REANIMATOR, a versatile framework designed to enable the repurposing of existing test collections by enriching them with extracted and synthetic resources. REANIMATOR enhances test collections from PDF files by parsing full texts and machine-readable tables, as well as related contextual information. It then employs state-of-the-art large language models to produce synthetic relevance labels. Including an optional human-in-the-loop step can help validate the resources that have been extracted and generated. We demonstrate its potential with a revitalized version of the TREC-COVID test collection, showcasing the development of a retrieval-augmented generation system and evaluating the impact of tables on retrieval-augmented generation. REANIMATOR enables the reuse of test collections for new applications, lowering costs and broadening the utility of legacy resources. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07584 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources Engelmann, Björn Haak, Fabian Schaer, Philipp Abdoust, Mani Erfanian Netze, Linus Bittkowski, Meik Information Retrieval Retrieval test collections are essential for evaluating information retrieval systems, yet they often lack generalizability across tasks. To overcome this limitation, we introduce REANIMATOR, a versatile framework designed to enable the repurposing of existing test collections by enriching them with extracted and synthetic resources. REANIMATOR enhances test collections from PDF files by parsing full texts and machine-readable tables, as well as related contextual information. It then employs state-of-the-art large language models to produce synthetic relevance labels. Including an optional human-in-the-loop step can help validate the resources that have been extracted and generated. We demonstrate its potential with a revitalized version of the TREC-COVID test collection, showcasing the development of a retrieval-augmented generation system and evaluating the impact of tables on retrieval-augmented generation. REANIMATOR enables the reuse of test collections for new applications, lowering costs and broadening the utility of legacy resources. |
| title | REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2504.07584 |