HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913509354242048 |
|---|---|
| author | Acosta, Julián N. Zhang, Xiaoman Dogra, Siddhant Zhou, Hong-Yu Payabvash, Seyedmehdi Falcone, Guido J. Oermann, Eric K. Rajpurkar, Pranav |
| author_facet | Acosta, Julián N. Zhang, Xiaoman Dogra, Siddhant Zhou, Hong-Yu Payabvash, Seyedmehdi Falcone, Guido J. Oermann, Eric K. Rajpurkar, Pranav |
| contents | We present Head CT Ontology Normalized Evaluation (HeadCT-ONE), a metric for evaluating head CT report generation through ontology-normalized entity and relation extraction. HeadCT-ONE enhances current information extraction derived metrics (such as RadGraph F1) by implementing entity normalization through domain-specific ontologies, addressing radiological language variability. HeadCT-ONE compares normalized entities and relations, allowing for controllable weighting of different entity types or specific entities. Through experiments on head CT reports from three health systems, we show that HeadCT-ONE's normalization and weighting approach improves the capture of semantically equivalent reports, better distinguishes between normal and abnormal reports, and aligns with radiologists' assessment of clinically significant errors, while offering flexibility to prioritize specific aspects of report content. Our results demonstrate how HeadCT-ONE enables more flexible, controllable, and granular automated evaluation of head CT reports. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_13038 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generation Acosta, Julián N. Zhang, Xiaoman Dogra, Siddhant Zhou, Hong-Yu Payabvash, Seyedmehdi Falcone, Guido J. Oermann, Eric K. Rajpurkar, Pranav Artificial Intelligence We present Head CT Ontology Normalized Evaluation (HeadCT-ONE), a metric for evaluating head CT report generation through ontology-normalized entity and relation extraction. HeadCT-ONE enhances current information extraction derived metrics (such as RadGraph F1) by implementing entity normalization through domain-specific ontologies, addressing radiological language variability. HeadCT-ONE compares normalized entities and relations, allowing for controllable weighting of different entity types or specific entities. Through experiments on head CT reports from three health systems, we show that HeadCT-ONE's normalization and weighting approach improves the capture of semantically equivalent reports, better distinguishes between normal and abnormal reports, and aligns with radiologists' assessment of clinically significant errors, while offering flexibility to prioritize specific aspects of report content. Our results demonstrate how HeadCT-ONE enables more flexible, controllable, and granular automated evaluation of head CT reports. |
| title | HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2409.13038 |