ReXamine-Global: A Framework for Uncovering Inconsistencies in Radiology Report Generation Metrics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916373874081792 |
|---|---|
| author | Banerjee, Oishi Saenz, Agustina Wu, Kay Clements, Warren Zia, Adil Buensalido, Dominic Kavnoudias, Helen Abi-Ghanem, Alain S. Ghawi, Nour El Luna, Cibele Castillo, Patricia Al-Surimi, Khaled Daghistani, Rayyan A. Chen, Yuh-Min Chao, Heng-sheng Heiliger, Lars Kim, Moon Haubold, Johannes Jonske, Frederic Rajpurkar, Pranav |
| author_facet | Banerjee, Oishi Saenz, Agustina Wu, Kay Clements, Warren Zia, Adil Buensalido, Dominic Kavnoudias, Helen Abi-Ghanem, Alain S. Ghawi, Nour El Luna, Cibele Castillo, Patricia Al-Surimi, Khaled Daghistani, Rayyan A. Chen, Yuh-Min Chao, Heng-sheng Heiliger, Lars Kim, Moon Haubold, Johannes Jonske, Frederic Rajpurkar, Pranav |
| contents | Given the rapidly expanding capabilities of generative AI models for radiology, there is a need for robust metrics that can accurately measure the quality of AI-generated radiology reports across diverse hospitals. We develop ReXamine-Global, a LLM-powered, multi-site framework that tests metrics across different writing styles and patient populations, exposing gaps in their generalization. First, our method tests whether a metric is undesirably sensitive to reporting style, providing different scores depending on whether AI-generated reports are stylistically similar to ground-truth reports or not. Second, our method measures whether a metric reliably agrees with experts, or whether metric and expert scores of AI-generated report quality diverge for some sites. Using 240 reports from 6 hospitals around the world, we apply ReXamine-Global to 7 established report evaluation metrics and uncover serious gaps in their generalizability. Developers can apply ReXamine-Global when designing new report evaluation metrics, ensuring their robustness across sites. Additionally, our analysis of existing metrics can guide users of those metrics towards evaluation procedures that work reliably at their sites of interest. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16208 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ReXamine-Global: A Framework for Uncovering Inconsistencies in Radiology Report Generation Metrics Banerjee, Oishi Saenz, Agustina Wu, Kay Clements, Warren Zia, Adil Buensalido, Dominic Kavnoudias, Helen Abi-Ghanem, Alain S. Ghawi, Nour El Luna, Cibele Castillo, Patricia Al-Surimi, Khaled Daghistani, Rayyan A. Chen, Yuh-Min Chao, Heng-sheng Heiliger, Lars Kim, Moon Haubold, Johannes Jonske, Frederic Rajpurkar, Pranav Machine Learning Computation and Language Given the rapidly expanding capabilities of generative AI models for radiology, there is a need for robust metrics that can accurately measure the quality of AI-generated radiology reports across diverse hospitals. We develop ReXamine-Global, a LLM-powered, multi-site framework that tests metrics across different writing styles and patient populations, exposing gaps in their generalization. First, our method tests whether a metric is undesirably sensitive to reporting style, providing different scores depending on whether AI-generated reports are stylistically similar to ground-truth reports or not. Second, our method measures whether a metric reliably agrees with experts, or whether metric and expert scores of AI-generated report quality diverge for some sites. Using 240 reports from 6 hospitals around the world, we apply ReXamine-Global to 7 established report evaluation metrics and uncover serious gaps in their generalizability. Developers can apply ReXamine-Global when designing new report evaluation metrics, ensuring their robustness across sites. Additionally, our analysis of existing metrics can guide users of those metrics towards evaluation procedures that work reliably at their sites of interest. |
| title | ReXamine-Global: A Framework for Uncovering Inconsistencies in Radiology Report Generation Metrics |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2408.16208 |