CURE: A Dataset for Clinical Understanding & Retrieval Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915363179986944 |
|---|---|
| author | Sheikh, Nadia Athar Marcos, Daniel Buades Jousse, Anne-Laure Oladipo, Akintunde Rousseau, Olivier Lin, Jimmy |
| author_facet | Sheikh, Nadia Athar Marcos, Daniel Buades Jousse, Anne-Laure Oladipo, Akintunde Rousseau, Olivier Lin, Jimmy |
| contents | Given the dominance of dense retrievers that do not generalize well beyond their training dataset distributions, domain-specific test sets are essential in evaluating retrieval. There are few test datasets for retrieval systems intended for use by healthcare providers in a point-of-care setting. To fill this gap we have collaborated with medical professionals to create CURE, an ad-hoc retrieval test dataset for passage ranking with 2000 queries spanning 10 medical domains with a monolingual (English) and two cross-lingual (French/Spanish -> English) conditions. In this paper, we describe how CURE was constructed and provide baseline results to showcase its effectiveness as an evaluation tool. CURE is published with a Creative Commons Attribution Non Commercial 4.0 license and can be accessed on Hugging Face and as a retrieval task on MTEB. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_06954 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CURE: A Dataset for Clinical Understanding & Retrieval Evaluation Sheikh, Nadia Athar Marcos, Daniel Buades Jousse, Anne-Laure Oladipo, Akintunde Rousseau, Olivier Lin, Jimmy Information Retrieval Given the dominance of dense retrievers that do not generalize well beyond their training dataset distributions, domain-specific test sets are essential in evaluating retrieval. There are few test datasets for retrieval systems intended for use by healthcare providers in a point-of-care setting. To fill this gap we have collaborated with medical professionals to create CURE, an ad-hoc retrieval test dataset for passage ranking with 2000 queries spanning 10 medical domains with a monolingual (English) and two cross-lingual (French/Spanish -> English) conditions. In this paper, we describe how CURE was constructed and provide baseline results to showcase its effectiveness as an evaluation tool. CURE is published with a Creative Commons Attribution Non Commercial 4.0 license and can be accessed on Hugging Face and as a retrieval task on MTEB. |
| title | CURE: A Dataset for Clinical Understanding & Retrieval Evaluation |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2412.06954 |