GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909554651955200 |
|---|---|
| author | Li, Tianbin Su, Yanzhou Li, Wei Fu, Bin Chen, Zhe Huang, Ziyan Wang, Guoan Ma, Chenglong Chen, Ying Hu, Ming Li, Yanjun Chen, Pengcheng Hu, Xiaowei Deng, Zhongying Ji, Yuanfeng Ye, Jin Qiao, Yu He, Junjun |
| author_facet | Li, Tianbin Su, Yanzhou Li, Wei Fu, Bin Chen, Zhe Huang, Ziyan Wang, Guoan Ma, Chenglong Chen, Ying Hu, Ming Li, Yanjun Chen, Pengcheng Hu, Xiaowei Deng, Zhongying Ji, Yuanfeng Ye, Jin Qiao, Yu He, Junjun |
| contents | Despite significant advancements in general AI, its effectiveness in the medical domain is limited by the lack of specialized medical knowledge. To address this, we formulate GMAI-VL-5.5M, a multimodal medical dataset created by converting hundreds of specialized medical datasets with various annotations into high-quality image-text pairs. This dataset offers comprehensive task coverage, diverse modalities, and rich image-text data. Building upon this dataset, we develop GMAI-VL, a general medical vision-language model, with a three-stage training strategy that enhances the integration of visual and textual information. This approach significantly improves the model's ability to process multimodal data, supporting accurate diagnoses and clinical decision-making. Experiments show that GMAI-VL achieves state-of-the-art performance across various multimodal medical tasks, including visual question answering and medical image diagnosis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_14522 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI Li, Tianbin Su, Yanzhou Li, Wei Fu, Bin Chen, Zhe Huang, Ziyan Wang, Guoan Ma, Chenglong Chen, Ying Hu, Ming Li, Yanjun Chen, Pengcheng Hu, Xiaowei Deng, Zhongying Ji, Yuanfeng Ye, Jin Qiao, Yu He, Junjun Computer Vision and Pattern Recognition Despite significant advancements in general AI, its effectiveness in the medical domain is limited by the lack of specialized medical knowledge. To address this, we formulate GMAI-VL-5.5M, a multimodal medical dataset created by converting hundreds of specialized medical datasets with various annotations into high-quality image-text pairs. This dataset offers comprehensive task coverage, diverse modalities, and rich image-text data. Building upon this dataset, we develop GMAI-VL, a general medical vision-language model, with a three-stage training strategy that enhances the integration of visual and textual information. This approach significantly improves the model's ability to process multimodal data, supporting accurate diagnoses and clinical decision-making. Experiments show that GMAI-VL achieves state-of-the-art performance across various multimodal medical tasks, including visual question answering and medical image diagnosis. |
| title | GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.14522 |