Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2601.09053 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911398800392192 |
|---|---|
| author | Li, Haiyi Li, Yutong Chi, Yiheng Deslandes, Alison Leonardi, Mathew Freger, Shay Zhang, Yuan Avery, Jodie Hull, M. Louise Chen, Hsiang-Ting |
| author_facet | Li, Haiyi Li, Yutong Chi, Yiheng Deslandes, Alison Leonardi, Mathew Freger, Shay Zhang, Yuan Avery, Jodie Hull, M. Louise Chen, Hsiang-Ting |
| contents | In this study, we evaluate a locally-deployed large-language model (LLM) to convert unstructured endometriosis transvaginal ultrasound (eTVUS) scan reports into structured data for imaging informatics workflows. Across 49 eTVUS reports, we compared three LLMs (7B/8B and a 20B-parameter model) against expert human extraction. The 20B model achieved a mean accuracy of 86.02%, substantially outperforming smaller models and confirming the importance of scale in handling complex clinical text. Crucially, we identified a highly complementary error profile: the LLM excelled at syntactic consistency (e.g., date/numeric formatting) where humans faltered, while human experts provided superior semantic and contextual interpretation. We also found that the LLM's semantic errors were fundamental limitations that could not be mitigated by simple prompt engineering. These findings strongly support a human-in-the-loop (HITL) workflow in which the on-premise LLM serves as a collaborative tool, not a full replacement. It automates routine structuring and flags potential human errors, enabling imaging specialists to focus on high-level semantic validation. We discuss implications for structured reporting and interactive AI systems in clinical practice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09053 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Who Fails Where? LLM and Human Error Patterns in Endometriosis Ultrasound Report Extraction Li, Haiyi Li, Yutong Chi, Yiheng Deslandes, Alison Leonardi, Mathew Freger, Shay Zhang, Yuan Avery, Jodie Hull, M. Louise Chen, Hsiang-Ting Human-Computer Interaction In this study, we evaluate a locally-deployed large-language model (LLM) to convert unstructured endometriosis transvaginal ultrasound (eTVUS) scan reports into structured data for imaging informatics workflows. Across 49 eTVUS reports, we compared three LLMs (7B/8B and a 20B-parameter model) against expert human extraction. The 20B model achieved a mean accuracy of 86.02%, substantially outperforming smaller models and confirming the importance of scale in handling complex clinical text. Crucially, we identified a highly complementary error profile: the LLM excelled at syntactic consistency (e.g., date/numeric formatting) where humans faltered, while human experts provided superior semantic and contextual interpretation. We also found that the LLM's semantic errors were fundamental limitations that could not be mitigated by simple prompt engineering. These findings strongly support a human-in-the-loop (HITL) workflow in which the on-premise LLM serves as a collaborative tool, not a full replacement. It automates routine structuring and flags potential human errors, enabling imaging specialists to focus on high-level semantic validation. We discuss implications for structured reporting and interactive AI systems in clinical practice. |
| title | Who Fails Where? LLM and Human Error Patterns in Endometriosis Ultrasound Report Extraction |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2601.09053 |