Zero-shot Generative Large Language Models for Systematic Review Screening Automation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909089722793984 |
|---|---|
| author | Wang, Shuai Scells, Harrisen Zhuang, Shengyao Potthast, Martin Koopman, Bevan Zuccon, Guido |
| author_facet | Wang, Shuai Scells, Harrisen Zhuang, Shengyao Potthast, Martin Koopman, Bevan Zuccon, Guido |
| contents | Systematic reviews are crucial for evidence-based medicine as they comprehensively analyse published research findings on specific questions. Conducting such reviews is often resource- and time-intensive, especially in the screening phase, where abstracts of publications are assessed for inclusion in a review. This study investigates the effectiveness of using zero-shot large language models~(LLMs) for automatic screening. We evaluate the effectiveness of eight different LLMs and investigate a calibration technique that uses a predefined recall threshold to determine whether a publication should be included in a systematic review. Our comprehensive evaluation using five standard test collections shows that instruction fine-tuning plays an important role in screening, that calibration renders LLMs practical for achieving a targeted recall, and that combining both with an ensemble of zero-shot models saves significant screening time compared to state-of-the-art approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_06320 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Zero-shot Generative Large Language Models for Systematic Review Screening Automation Wang, Shuai Scells, Harrisen Zhuang, Shengyao Potthast, Martin Koopman, Bevan Zuccon, Guido Information Retrieval Computation and Language Systematic reviews are crucial for evidence-based medicine as they comprehensively analyse published research findings on specific questions. Conducting such reviews is often resource- and time-intensive, especially in the screening phase, where abstracts of publications are assessed for inclusion in a review. This study investigates the effectiveness of using zero-shot large language models~(LLMs) for automatic screening. We evaluate the effectiveness of eight different LLMs and investigate a calibration technique that uses a predefined recall threshold to determine whether a publication should be included in a systematic review. Our comprehensive evaluation using five standard test collections shows that instruction fine-tuning plays an important role in screening, that calibration renders LLMs practical for achieving a targeted recall, and that combining both with an ensemble of zero-shot models saves significant screening time compared to state-of-the-art approaches. |
| title | Zero-shot Generative Large Language Models for Systematic Review Screening Automation |
| topic | Information Retrieval Computation and Language |
| url | https://arxiv.org/abs/2401.06320 |