Ultrasound Report Generation with Cross-Modality Feature Alignment via Unsupervised Guidance
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911898991067136 |
|---|---|
| author | Li, Jun Su, Tongkun Zhao, Baoliang Lv, Faqin Wang, Qiong Navab, Nassir Hu, Ying Jiang, Zhongliang |
| author_facet | Li, Jun Su, Tongkun Zhao, Baoliang Lv, Faqin Wang, Qiong Navab, Nassir Hu, Ying Jiang, Zhongliang |
| contents | Automatic report generation has arisen as a significant research area in computer-aided diagnosis, aiming to alleviate the burden on clinicians by generating reports automatically based on medical images. In this work, we propose a novel framework for automatic ultrasound report generation, leveraging a combination of unsupervised and supervised learning methods to aid the report generation process. Our framework incorporates unsupervised learning methods to extract potential knowledge from ultrasound text reports, serving as the prior information to guide the model in aligning visual and textual features, thereby addressing the challenge of feature discrepancy. Additionally, we design a global semantic comparison mechanism to enhance the performance of generating more comprehensive and accurate medical reports. To enable the implementation of ultrasound report generation, we constructed three large-scale ultrasound image-text datasets from different organs for training and validation purposes. Extensive evaluations with other state-of-the-art approaches exhibit its superior performance across all three datasets. Code and dataset are valuable at this link. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_00644 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Ultrasound Report Generation with Cross-Modality Feature Alignment via Unsupervised Guidance Li, Jun Su, Tongkun Zhao, Baoliang Lv, Faqin Wang, Qiong Navab, Nassir Hu, Ying Jiang, Zhongliang Computer Vision and Pattern Recognition Automatic report generation has arisen as a significant research area in computer-aided diagnosis, aiming to alleviate the burden on clinicians by generating reports automatically based on medical images. In this work, we propose a novel framework for automatic ultrasound report generation, leveraging a combination of unsupervised and supervised learning methods to aid the report generation process. Our framework incorporates unsupervised learning methods to extract potential knowledge from ultrasound text reports, serving as the prior information to guide the model in aligning visual and textual features, thereby addressing the challenge of feature discrepancy. Additionally, we design a global semantic comparison mechanism to enhance the performance of generating more comprehensive and accurate medical reports. To enable the implementation of ultrasound report generation, we constructed three large-scale ultrasound image-text datasets from different organs for training and validation purposes. Extensive evaluations with other state-of-the-art approaches exhibit its superior performance across all three datasets. Code and dataset are valuable at this link. |
| title | Ultrasound Report Generation with Cross-Modality Feature Alignment via Unsupervised Guidance |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.00644 |