Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916615515275264 |
|---|---|
| author | Deng, Qiao Huang, Zhongzhen Wang, Yunqi Wang, Zhichuan Wang, Zhao Zhang, Xiaofan Dou, Qi Hui, Yeung Yu Hui, Edward S. |
| author_facet | Deng, Qiao Huang, Zhongzhen Wang, Yunqi Wang, Zhichuan Wang, Zhao Zhang, Xiaofan Dou, Qi Hui, Yeung Yu Hui, Edward S. |
| contents | Medical foundation models have the potential to revolutionize healthcare by providing robust and generalized representations of medical data. Medical vision-language pre-training has emerged as a promising approach for learning domain-general representations of medical image and text. Current algorithms that exploit global and local alignment between medical image and text could however be marred by redundant information in medical data. To address this issue, we propose a grounded knowledge-enhanced medical vision-language pre-training (GK-MVLP) framework for chest X-ray. In this framework, medical knowledge was grounded to the appropriate anatomical regions by using a transformer-based grounded knowledge-enhanced module for fine-grained alignment between textural features of medical knowledge and the corresponding anatomical region-level visual features. The performance of GK-MVLP was competitive with or exceeded the state of the art on downstream image understanding tasks (chest X-ray disease classification, disease localization), generative task (report generation), and vision-language understanding task (medical visual question-answering). Our results demonstrate the advantage of incorporating grounding mechanism to remove biases and improve the alignment between chest X-ray image and radiology report. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_14750 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray Deng, Qiao Huang, Zhongzhen Wang, Yunqi Wang, Zhichuan Wang, Zhao Zhang, Xiaofan Dou, Qi Hui, Yeung Yu Hui, Edward S. Computer Vision and Pattern Recognition Artificial Intelligence Medical foundation models have the potential to revolutionize healthcare by providing robust and generalized representations of medical data. Medical vision-language pre-training has emerged as a promising approach for learning domain-general representations of medical image and text. Current algorithms that exploit global and local alignment between medical image and text could however be marred by redundant information in medical data. To address this issue, we propose a grounded knowledge-enhanced medical vision-language pre-training (GK-MVLP) framework for chest X-ray. In this framework, medical knowledge was grounded to the appropriate anatomical regions by using a transformer-based grounded knowledge-enhanced module for fine-grained alignment between textural features of medical knowledge and the corresponding anatomical region-level visual features. The performance of GK-MVLP was competitive with or exceeded the state of the art on downstream image understanding tasks (chest X-ray disease classification, disease localization), generative task (report generation), and vision-language understanding task (medical visual question-answering). Our results demonstrate the advantage of incorporating grounding mechanism to remove biases and improve the alignment between chest X-ray image and radiology report. |
| title | Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2404.14750 |