MRScore: Evaluating Radiology Report Generation with LLM-based Reward System
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916226910912512 |
|---|---|
| author | Liu, Yunyi Wang, Zhanyu Li, Yingshu Liang, Xinyu Liu, Lingqiao Wang, Lei Zhou, Luping |
| author_facet | Liu, Yunyi Wang, Zhanyu Li, Yingshu Liang, Xinyu Liu, Lingqiao Wang, Lei Zhou, Luping |
| contents | In recent years, automated radiology report generation has experienced significant growth. This paper introduces MRScore, an automatic evaluation metric tailored for radiology report generation by leveraging Large Language Models (LLMs). Conventional NLG (natural language generation) metrics like BLEU are inadequate for accurately assessing the generated radiology reports, as systematically demonstrated by our observations within this paper. To address this challenge, we collaborated with radiologists to develop a framework that guides LLMs for radiology report evaluation, ensuring alignment with human analysis. Our framework includes two key components: i) utilizing GPT to generate large amounts of training data, i.e., reports with different qualities, and ii) pairing GPT-generated reports as accepted and rejected samples and training LLMs to produce MRScore as the model reward. Our experiments demonstrate MRScore's higher correlation with human judgments and superior performance in model selection compared to traditional metrics. Our code and datasets will be available on GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_17778 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MRScore: Evaluating Radiology Report Generation with LLM-based Reward System Liu, Yunyi Wang, Zhanyu Li, Yingshu Liang, Xinyu Liu, Lingqiao Wang, Lei Zhou, Luping Computation and Language Artificial Intelligence In recent years, automated radiology report generation has experienced significant growth. This paper introduces MRScore, an automatic evaluation metric tailored for radiology report generation by leveraging Large Language Models (LLMs). Conventional NLG (natural language generation) metrics like BLEU are inadequate for accurately assessing the generated radiology reports, as systematically demonstrated by our observations within this paper. To address this challenge, we collaborated with radiologists to develop a framework that guides LLMs for radiology report evaluation, ensuring alignment with human analysis. Our framework includes two key components: i) utilizing GPT to generate large amounts of training data, i.e., reports with different qualities, and ii) pairing GPT-generated reports as accepted and rejected samples and training LLMs to produce MRScore as the model reward. Our experiments demonstrate MRScore's higher correlation with human judgments and superior performance in model selection compared to traditional metrics. Our code and datasets will be available on GitHub. |
| title | MRScore: Evaluating Radiology Report Generation with LLM-based Reward System |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2404.17778 |