Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.22935 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918406260785152 |
|---|---|
| author | Zhang, Ran Lin, Yucong Su, Zhaoli Liu, Bowen Ai, Danni Fu, Tianyu Xiao, Deqiang Fan, Jingfan Wang, Yuanyuan Gao, Mingwei Hu, Yuwan Gao, Shuya Li, Jingtao Yang, Jian Song, Hong Sun, Hongliang |
| author_facet | Zhang, Ran Lin, Yucong Su, Zhaoli Liu, Bowen Ai, Danni Fu, Tianyu Xiao, Deqiang Fan, Jingfan Wang, Yuanyuan Gao, Mingwei Hu, Yuwan Gao, Shuya Li, Jingtao Yang, Jian Song, Hong Sun, Hongliang |
| contents | Chest X-ray report generation and automated evaluation are limited by poor recognition of low-prevalence abnormalities and inadequate handling of clinically important language, including negation and ambiguity. We develop a clinician-guided framework combining human expertise and large language models for multi-label finding extraction from free-text chest X-ray reports and use it to define Ran Score, a finding-level metric for report evaluation. Using three non-overlapping MIMIC-CXR-EN cohorts from a public chest X-ray dataset and an independent ChestX-CN validation cohort, we optimize prompts, establish radiologist-derived reference labels and evaluate report generation models. The optimized framework improves the macro-averaged score from 0.753 to 0.956 on the MIMIC-CXR-EN development cohort, exceeds the CheXbert benchmark by 15.7 percentage points on directly comparable labels, and shows robust generalization on the ChestX-CN validation cohort. Here we show that clinician-guided prompt optimization improves agreement with a radiologist-derived reference standard and that Ran Score enables finding-level evaluation of report fidelity, particularly for low-prevalence abnormalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_22935 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Ran Score: a LLM-based Evaluation Score for Radiology Report Generation Zhang, Ran Lin, Yucong Su, Zhaoli Liu, Bowen Ai, Danni Fu, Tianyu Xiao, Deqiang Fan, Jingfan Wang, Yuanyuan Gao, Mingwei Hu, Yuwan Gao, Shuya Li, Jingtao Yang, Jian Song, Hong Sun, Hongliang Artificial Intelligence Human-Computer Interaction Chest X-ray report generation and automated evaluation are limited by poor recognition of low-prevalence abnormalities and inadequate handling of clinically important language, including negation and ambiguity. We develop a clinician-guided framework combining human expertise and large language models for multi-label finding extraction from free-text chest X-ray reports and use it to define Ran Score, a finding-level metric for report evaluation. Using three non-overlapping MIMIC-CXR-EN cohorts from a public chest X-ray dataset and an independent ChestX-CN validation cohort, we optimize prompts, establish radiologist-derived reference labels and evaluate report generation models. The optimized framework improves the macro-averaged score from 0.753 to 0.956 on the MIMIC-CXR-EN development cohort, exceeds the CheXbert benchmark by 15.7 percentage points on directly comparable labels, and shows robust generalization on the ChestX-CN validation cohort. Here we show that clinician-guided prompt optimization improves agreement with a radiologist-derived reference standard and that Ran Score enables finding-level evaluation of report fidelity, particularly for low-prevalence abnormalities. |
| title | Ran Score: a LLM-based Evaluation Score for Radiology Report Generation |
| topic | Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2603.22935 |