Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918264871845888 |
|---|---|
| author | Xue, Dong Shao, Ziyao Duan, Zhaoyang Liu, Fangzhou Li, Bing Zhang, Zhongheng |
| author_facet | Xue, Dong Shao, Ziyao Duan, Zhaoyang Liu, Fangzhou Li, Bing Zhang, Zhongheng |
| contents | Large multimodal models (LMMs) have demonstrated significant potential in providing innovative solutions for various biomedical tasks, including pathology analysis, radiology report generation, and biomedical assistance. However, the existing multimodal biomedical AI is typically based on foundation LLMs, thus hindering the understanding of intricate medical concepts with limited medical training data. Moreover, recent LLaVA-induced medical LMMs struggle to effectively capture the intricate relationship between the texts and the images. Therefore, we introduce Doctor Sun, a large multimodal generative model specialized in medicine, developed to encode, integrate, and interpret diverse biomedical data modalities such as text and images. In particular, Doctor Sun integrates a pre-trained vision encoder with a medical LLM and conducts two-stage training on various medical datasets, focusing on feature alignment and instruction tuning. Moreover, we release SunMed-VL, a wide-range bilingual medical multimodal dataset, along with all associated models, code, and resources, to freely support the advancement of biomedical multimodal research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_08270 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI Xue, Dong Shao, Ziyao Duan, Zhaoyang Liu, Fangzhou Li, Bing Zhang, Zhongheng Machine Learning Artificial Intelligence Computation and Language Multimedia Large multimodal models (LMMs) have demonstrated significant potential in providing innovative solutions for various biomedical tasks, including pathology analysis, radiology report generation, and biomedical assistance. However, the existing multimodal biomedical AI is typically based on foundation LLMs, thus hindering the understanding of intricate medical concepts with limited medical training data. Moreover, recent LLaVA-induced medical LMMs struggle to effectively capture the intricate relationship between the texts and the images. Therefore, we introduce Doctor Sun, a large multimodal generative model specialized in medicine, developed to encode, integrate, and interpret diverse biomedical data modalities such as text and images. In particular, Doctor Sun integrates a pre-trained vision encoder with a medical LLM and conducts two-stage training on various medical datasets, focusing on feature alignment and instruction tuning. Moreover, we release SunMed-VL, a wide-range bilingual medical multimodal dataset, along with all associated models, code, and resources, to freely support the advancement of biomedical multimodal research. |
| title | Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI |
| topic | Machine Learning Artificial Intelligence Computation and Language Multimedia |
| url | https://arxiv.org/abs/2508.08270 |