GREEN: Generative Radiology Report Evaluation and Error Notation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909463531749376 |
|---|---|
| author | Ostmeier, Sophie Xu, Justin Chen, Zhihong Varma, Maya Blankemeier, Louis Bluethgen, Christian Michalson, Arne Edward Moseley, Michael Langlotz, Curtis Chaudhari, Akshay S Delbrouck, Jean-Benoit |
| author_facet | Ostmeier, Sophie Xu, Justin Chen, Zhihong Varma, Maya Blankemeier, Louis Bluethgen, Christian Michalson, Arne Edward Moseley, Michael Langlotz, Curtis Chaudhari, Akshay S Delbrouck, Jean-Benoit |
| contents | Evaluating radiology reports is a challenging problem as factual correctness is extremely important due to the need for accurate medical communication about medical images. Existing automatic evaluation metrics either suffer from failing to consider factual correctness (e.g., BLEU and ROUGE) or are limited in their interpretability (e.g., F1CheXpert and F1RadGraph). In this paper, we introduce GREEN (Generative Radiology Report Evaluation and Error Notation), a radiology report generation metric that leverages the natural language understanding of language models to identify and explain clinically significant errors in candidate reports, both quantitatively and qualitatively. Compared to current metrics, GREEN offers: 1) a score aligned with expert preferences, 2) human interpretable explanations of clinically significant errors, enabling feedback loops with end-users, and 3) a lightweight open-source method that reaches the performance of commercial counterparts. We validate our GREEN metric by comparing it to GPT-4, as well as to error counts of 6 experts and preferences of 2 experts. Our method demonstrates not only higher correlation with expert error counts, but simultaneously higher alignment with expert preferences when compared to previous approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_03595 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GREEN: Generative Radiology Report Evaluation and Error Notation Ostmeier, Sophie Xu, Justin Chen, Zhihong Varma, Maya Blankemeier, Louis Bluethgen, Christian Michalson, Arne Edward Moseley, Michael Langlotz, Curtis Chaudhari, Akshay S Delbrouck, Jean-Benoit Computation and Language Artificial Intelligence Evaluating radiology reports is a challenging problem as factual correctness is extremely important due to the need for accurate medical communication about medical images. Existing automatic evaluation metrics either suffer from failing to consider factual correctness (e.g., BLEU and ROUGE) or are limited in their interpretability (e.g., F1CheXpert and F1RadGraph). In this paper, we introduce GREEN (Generative Radiology Report Evaluation and Error Notation), a radiology report generation metric that leverages the natural language understanding of language models to identify and explain clinically significant errors in candidate reports, both quantitatively and qualitatively. Compared to current metrics, GREEN offers: 1) a score aligned with expert preferences, 2) human interpretable explanations of clinically significant errors, enabling feedback loops with end-users, and 3) a lightweight open-source method that reaches the performance of commercial counterparts. We validate our GREEN metric by comparing it to GPT-4, as well as to error counts of 6 experts and preferences of 2 experts. Our method demonstrates not only higher correlation with expert error counts, but simultaneously higher alignment with expert preferences when compared to previous approaches. |
| title | GREEN: Generative Radiology Report Evaluation and Error Notation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.03595 |