Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.00998 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910393749733376 |
|---|---|
| author | Wang, Zilong Luo, Xufang Jiang, Xinyang Li, Dongsheng Qiu, Lili |
| author_facet | Wang, Zilong Luo, Xufang Jiang, Xinyang Li, Dongsheng Qiu, Lili |
| contents | Evaluating generated radiology reports is crucial for the development of radiology AI, but existing metrics fail to reflect the task's clinical requirements. This study proposes a novel evaluation framework using large language models (LLMs) to compare radiology reports for assessment. We compare the performance of various LLMs and demonstrate that, when using GPT-4, our proposed metric achieves evaluation consistency close to that of radiologists. Furthermore, to reduce costs and improve accessibility, making this method practical, we construct a dataset using LLM evaluation results and perform knowledge distillation to train a smaller model. The distilled model achieves evaluation capabilities comparable to GPT-4. Our framework and distilled model offer an accessible and efficient evaluation method for radiology report generation, facilitating the development of more clinically relevant models. The model will be further open-sourced and accessible. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_00998 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation Wang, Zilong Luo, Xufang Jiang, Xinyang Li, Dongsheng Qiu, Lili Computation and Language Artificial Intelligence Evaluating generated radiology reports is crucial for the development of radiology AI, but existing metrics fail to reflect the task's clinical requirements. This study proposes a novel evaluation framework using large language models (LLMs) to compare radiology reports for assessment. We compare the performance of various LLMs and demonstrate that, when using GPT-4, our proposed metric achieves evaluation consistency close to that of radiologists. Furthermore, to reduce costs and improve accessibility, making this method practical, we construct a dataset using LLM evaluation results and perform knowledge distillation to train a smaller model. The distilled model achieves evaluation capabilities comparable to GPT-4. Our framework and distilled model offer an accessible and efficient evaluation method for radiology report generation, facilitating the development of more clinically relevant models. The model will be further open-sourced and accessible. |
| title | LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2404.00998 |