Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909721807552512 |
|---|---|
| author | Dua, Radhika Joon, Young Kwon Dogra, Siddhant Freedman, Daniel Ruan, Diana Nashawaty, Motaz Rigau, Danielle Alber, Daniel Alexander Zhang, Kang Cho, Kyunghyun Oermann, Eric Karl |
| author_facet | Dua, Radhika Joon, Young Kwon Dogra, Siddhant Freedman, Daniel Ruan, Diana Nashawaty, Motaz Rigau, Danielle Alber, Daniel Alexander Zhang, Kang Cho, Kyunghyun Oermann, Eric Karl |
| contents | Radiological imaging is central to diagnosis, treatment planning, and clinical decision-making. Vision-language foundation models have spurred interest in automated radiology report generation (RRG), but safe deployment requires reliable clinical evaluation of generated reports. Existing metrics often rely on surface-level similarity or behave as black boxes, lacking interpretability. We introduce ICARE (Interpretable and Clinically-grounded Agent-based Report Evaluation), an interpretable evaluation framework leveraging large language model agents and dynamic multiple-choice question answering (MCQA). Two agents, each with either the ground-truth or generated report, generate clinically meaningful questions and quiz each other. Agreement on answers captures preservation and consistency of findings, serving as interpretable proxies for clinical precision and recall. By linking scores to question-answer pairs, ICARE enables transparent, and interpretable assessment. Clinician studies show ICARE aligns significantly more with expert judgment than prior metrics. Perturbation analyses confirm sensitivity to clinical content and reproducibility, while model comparisons reveal interpretable error patterns. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_02808 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation Dua, Radhika Joon, Young Kwon Dogra, Siddhant Freedman, Daniel Ruan, Diana Nashawaty, Motaz Rigau, Danielle Alber, Daniel Alexander Zhang, Kang Cho, Kyunghyun Oermann, Eric Karl Computation and Language Artificial Intelligence Machine Learning Radiological imaging is central to diagnosis, treatment planning, and clinical decision-making. Vision-language foundation models have spurred interest in automated radiology report generation (RRG), but safe deployment requires reliable clinical evaluation of generated reports. Existing metrics often rely on surface-level similarity or behave as black boxes, lacking interpretability. We introduce ICARE (Interpretable and Clinically-grounded Agent-based Report Evaluation), an interpretable evaluation framework leveraging large language model agents and dynamic multiple-choice question answering (MCQA). Two agents, each with either the ground-truth or generated report, generate clinically meaningful questions and quiz each other. Agreement on answers captures preservation and consistency of findings, serving as interpretable proxies for clinical precision and recall. By linking scores to question-answer pairs, ICARE enables transparent, and interpretable assessment. Clinician studies show ICARE aligns significantly more with expert judgment than prior metrics. Perturbation analyses confirm sensitivity to clinical content and reproducibility, while model comparisons reveal interpretable error patterns. |
| title | Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2508.02808 |