Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.18736 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918118121537536 |
|---|---|
| author | Wang, Jianyou Cao, Weili Wang, Kaicheng Wang, Xiaoyue Dalvi, Ashish Prasad, Gino Liang, Qishan Her, Hsuan-lin Wang, Ming Yang, Qin Yeo, Gene W. Neal, David E. Khan, Maxim Rosin, Christopher D. Paturi, Ramamohan Bergen, Leon |
| author_facet | Wang, Jianyou Cao, Weili Wang, Kaicheng Wang, Xiaoyue Dalvi, Ashish Prasad, Gino Liang, Qishan Her, Hsuan-lin Wang, Ming Yang, Qin Yeo, Gene W. Neal, David E. Khan, Maxim Rosin, Christopher D. Paturi, Ramamohan Bergen, Leon |
| contents | We study the task of automatically finding evidence relevant to hypotheses in biomedical papers. Finding relevant evidence is an important step when researchers investigate scientific hypotheses. We introduce EvidenceBench to measure models performance on this task, which is created by a novel pipeline that consists of hypothesis generation and sentence-by-sentence annotation of biomedical papers for relevant evidence, completely guided by and faithfully following existing human experts judgment. We demonstrate the pipeline's validity and accuracy with multiple sets of human-expert annotations. We evaluated a diverse set of language models and retrieval systems on the benchmark and found that model performances still fall significantly short of the expert level on this task. To show the scalability of our proposed pipeline, we create a larger EvidenceBench-100k with 107,461 fully annotated papers with hypotheses to facilitate model training and development. Both datasets are available at https://github.com/EvidenceBench/EvidenceBench |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_18736 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers Wang, Jianyou Cao, Weili Wang, Kaicheng Wang, Xiaoyue Dalvi, Ashish Prasad, Gino Liang, Qishan Her, Hsuan-lin Wang, Ming Yang, Qin Yeo, Gene W. Neal, David E. Khan, Maxim Rosin, Christopher D. Paturi, Ramamohan Bergen, Leon Computation and Language We study the task of automatically finding evidence relevant to hypotheses in biomedical papers. Finding relevant evidence is an important step when researchers investigate scientific hypotheses. We introduce EvidenceBench to measure models performance on this task, which is created by a novel pipeline that consists of hypothesis generation and sentence-by-sentence annotation of biomedical papers for relevant evidence, completely guided by and faithfully following existing human experts judgment. We demonstrate the pipeline's validity and accuracy with multiple sets of human-expert annotations. We evaluated a diverse set of language models and retrieval systems on the benchmark and found that model performances still fall significantly short of the expert level on this task. To show the scalability of our proposed pipeline, we create a larger EvidenceBench-100k with 107,461 fully annotated papers with hypotheses to facilitate model training and development. Both datasets are available at https://github.com/EvidenceBench/EvidenceBench |
| title | EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.18736 |