Ultrasound Report Generation with Multimodal Large Language Models for Standardized Texts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912380717367296 |
|---|---|
| author | Ge, Peixuan Su, Tongkun Lv, Faqin Zhao, Baoliang Zhang, Peng Wong, Chi Hong Yao, Liang Sun, Yu Wang, Zenan Wong, Pak Kin Hu, Ying |
| author_facet | Ge, Peixuan Su, Tongkun Lv, Faqin Zhao, Baoliang Zhang, Peng Wong, Chi Hong Yao, Liang Sun, Yu Wang, Zenan Wong, Pak Kin Hu, Ying |
| contents | Ultrasound (US) report generation is a challenging task due to the variability of US images, operator dependence, and the need for standardized text. Unlike X-ray and CT, US imaging lacks consistent datasets, making automation difficult. In this study, we propose a unified framework for multi-organ and multilingual US report generation, integrating fragment-based multilingual training and leveraging the standardized nature of US reports. By aligning modular text fragments with diverse imaging data and curating a bilingual English-Chinese dataset, the method achieves consistent and clinically accurate text generation across organ sites and languages. Fine-tuning with selective unfreezing of the vision transformer (ViT) further improves text-image alignment. Compared to the previous state-of-the-art KMVE method, our approach achieves relative gains of about 2\% in BLEU scores, approximately 3\% in ROUGE-L, and about 15\% in CIDEr, while significantly reducing errors such as missing or incorrect content. By unifying multi-organ and multi-language report generation into a single, scalable framework, this work demonstrates strong potential for real-world clinical workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_08838 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ultrasound Report Generation with Multimodal Large Language Models for Standardized Texts Ge, Peixuan Su, Tongkun Lv, Faqin Zhao, Baoliang Zhang, Peng Wong, Chi Hong Yao, Liang Sun, Yu Wang, Zenan Wong, Pak Kin Hu, Ying Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition Ultrasound (US) report generation is a challenging task due to the variability of US images, operator dependence, and the need for standardized text. Unlike X-ray and CT, US imaging lacks consistent datasets, making automation difficult. In this study, we propose a unified framework for multi-organ and multilingual US report generation, integrating fragment-based multilingual training and leveraging the standardized nature of US reports. By aligning modular text fragments with diverse imaging data and curating a bilingual English-Chinese dataset, the method achieves consistent and clinically accurate text generation across organ sites and languages. Fine-tuning with selective unfreezing of the vision transformer (ViT) further improves text-image alignment. Compared to the previous state-of-the-art KMVE method, our approach achieves relative gains of about 2\% in BLEU scores, approximately 3\% in ROUGE-L, and about 15\% in CIDEr, while significantly reducing errors such as missing or incorrect content. By unifying multi-organ and multi-language report generation into a single, scalable framework, this work demonstrates strong potential for real-world clinical workflows. |
| title | Ultrasound Report Generation with Multimodal Large Language Models for Standardized Texts |
| topic | Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.08838 |