Large language models for automated PRISMA 2020 adherence checking
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909915865415680 |
|---|---|
| author | Kataoka, Yuki So, Ryuhei Banno, Masahiro Tsujimoto, Yasushi Takayama, Tomohiro Yamagishi, Yosuke Tsuge, Takahiro Yamamoto, Norio Suda, Chiaki Furukawa, Toshi A. |
| author_facet | Kataoka, Yuki So, Ryuhei Banno, Masahiro Tsujimoto, Yasushi Takayama, Tomohiro Yamagishi, Yosuke Tsuge, Takahiro Yamamoto, Norio Suda, Chiaki Furukawa, Toshi A. |
| contents | Evaluating adherence to PRISMA 2020 guideline remains a burden in the peer review process. To address the lack of shareable benchmarks, we constructed a copyright-aware benchmark of 108 Creative Commons-licensed systematic reviews and evaluated ten large language models (LLMs) across five input formats. In a development cohort, supplying structured PRISMA 2020 checklists (Markdown, JSON, XML, or plain text) yielded 78.7-79.7% accuracy versus 45.21% for manuscript-only input (p less than 0.0001), with no differences between structured formats (p>0.9). Across models, accuracy ranged from 70.6-82.8% with distinct sensitivity-specificity trade-offs, replicated in an independent validation cohort. We then selected Qwen3-Max (a high-sensitivity open-weight model) and extended evaluation to the full dataset (n=120), achieving 95.1% sensitivity and 49.3% specificity. Structured checklist provision substantially improves LLM-based PRISMA assessment, though human expert verification remains essential before editorial decisions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_16707 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Large language models for automated PRISMA 2020 adherence checking Kataoka, Yuki So, Ryuhei Banno, Masahiro Tsujimoto, Yasushi Takayama, Tomohiro Yamagishi, Yosuke Tsuge, Takahiro Yamamoto, Norio Suda, Chiaki Furukawa, Toshi A. Software Engineering Artificial Intelligence Evaluating adherence to PRISMA 2020 guideline remains a burden in the peer review process. To address the lack of shareable benchmarks, we constructed a copyright-aware benchmark of 108 Creative Commons-licensed systematic reviews and evaluated ten large language models (LLMs) across five input formats. In a development cohort, supplying structured PRISMA 2020 checklists (Markdown, JSON, XML, or plain text) yielded 78.7-79.7% accuracy versus 45.21% for manuscript-only input (p less than 0.0001), with no differences between structured formats (p>0.9). Across models, accuracy ranged from 70.6-82.8% with distinct sensitivity-specificity trade-offs, replicated in an independent validation cohort. We then selected Qwen3-Max (a high-sensitivity open-weight model) and extended evaluation to the full dataset (n=120), achieving 95.1% sensitivity and 49.3% specificity. Structured checklist provision substantially improves LLM-based PRISMA assessment, though human expert verification remains essential before editorial decisions. |
| title | Large language models for automated PRISMA 2020 adherence checking |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2511.16707 |