Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.24231 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914063892611072 |
|---|---|
| author | Bai, Yang Cheng, Haoran Zhou, Yang Zhou, Jun Thirunavukarasu, Arun Ke, Yuhe Yao, Jie Fukutsu, Kanae Quek, Chrystie Wan Ning Hong, Ashley Gutierrez, Laura Teo, Zhen Ling Ting, Darren Shu Jeng Soetikno, Brian T. Nielsen, Christopher S. Elze, Tobias Li, Zengxiang Dinh, Linh Le Chan, Hiok Hong Koh, Victor Tan, Marcus Li, Kelvin Z. Yip, Leonard Cheng, Ching Yu Tham, Yih Chung Tan, Gavin Siew Wei Schmetterer, Leopold Ang, Marcus Hussain, Rahat Mehta, Jod Aung, Tin Cheng, Lionel Tim-Ee Anh, Tran Nguyen Tuan Cheng, Chee Leong Wong, Tien Yin Liu, Nan Tan, Iain Beehuat Lim, Soon Thye Klang, Eyal Lim, Tony Kiat Hon Goh, Rick Siow Mong Liu, Yong Ting, Daniel Shu Wei |
| author_facet | Bai, Yang Cheng, Haoran Zhou, Yang Zhou, Jun Thirunavukarasu, Arun Ke, Yuhe Yao, Jie Fukutsu, Kanae Quek, Chrystie Wan Ning Hong, Ashley Gutierrez, Laura Teo, Zhen Ling Ting, Darren Shu Jeng Soetikno, Brian T. Nielsen, Christopher S. Elze, Tobias Li, Zengxiang Dinh, Linh Le Chan, Hiok Hong Koh, Victor Tan, Marcus Li, Kelvin Z. Yip, Leonard Cheng, Ching Yu Tham, Yih Chung Tan, Gavin Siew Wei Schmetterer, Leopold Ang, Marcus Hussain, Rahat Mehta, Jod Aung, Tin Cheng, Lionel Tim-Ee Anh, Tran Nguyen Tuan Cheng, Chee Leong Wong, Tien Yin Liu, Nan Tan, Iain Beehuat Lim, Soon Thye Klang, Eyal Lim, Tony Kiat Hon Goh, Rick Siow Mong Liu, Yong Ting, Daniel Shu Wei |
| contents | Despite the promise of foundation models in medical AI, current systems remain limited - they are modality-specific and lack transparent reasoning processes, hindering clinical adoption. To address this gap, we present EVLF-FM, a multimodal vision-language foundation model (VLM) designed to unify broad diagnostic capability with fine-grain explainability. The development and testing of EVLF-FM encompassed over 1.3 million total samples from 23 global datasets across eleven imaging modalities related to six clinical specialties: dermatology, hepatology, ophthalmology, pathology, pulmonology, and radiology. External validation employed 8,884 independent test samples from 10 additional datasets across five imaging modalities. Technically, EVLF-FM is developed to assist with multiple disease diagnosis and visual question answering with pixel-level visual grounding and reasoning capabilities. In internal validation for disease diagnostics, EVLF-FM achieved the highest average accuracy (0.858) and F1-score (0.797), outperforming leading generalist and specialist models. In medical visual grounding, EVLF-FM also achieved stellar performance across nine modalities with average mIOU of 0.743 and Acc@0.5 of 0.837. External validations further confirmed strong zero-shot and few-shot performance, with competitive F1-scores despite a smaller model size. Through a hybrid training strategy combining supervised and visual reinforcement fine-tuning, EVLF-FM not only achieves state-of-the-art accuracy but also exhibits step-by-step reasoning, aligning outputs with visual evidence. EVLF-FM is an early multi-disease VLM model with explainability and reasoning capabilities that could advance adoption of and trust in foundation models for real-world clinical deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24231 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EVLF-FM: Explainable Vision Language Foundation Model for Medicine Bai, Yang Cheng, Haoran Zhou, Yang Zhou, Jun Thirunavukarasu, Arun Ke, Yuhe Yao, Jie Fukutsu, Kanae Quek, Chrystie Wan Ning Hong, Ashley Gutierrez, Laura Teo, Zhen Ling Ting, Darren Shu Jeng Soetikno, Brian T. Nielsen, Christopher S. Elze, Tobias Li, Zengxiang Dinh, Linh Le Chan, Hiok Hong Koh, Victor Tan, Marcus Li, Kelvin Z. Yip, Leonard Cheng, Ching Yu Tham, Yih Chung Tan, Gavin Siew Wei Schmetterer, Leopold Ang, Marcus Hussain, Rahat Mehta, Jod Aung, Tin Cheng, Lionel Tim-Ee Anh, Tran Nguyen Tuan Cheng, Chee Leong Wong, Tien Yin Liu, Nan Tan, Iain Beehuat Lim, Soon Thye Klang, Eyal Lim, Tony Kiat Hon Goh, Rick Siow Mong Liu, Yong Ting, Daniel Shu Wei Computer Vision and Pattern Recognition Despite the promise of foundation models in medical AI, current systems remain limited - they are modality-specific and lack transparent reasoning processes, hindering clinical adoption. To address this gap, we present EVLF-FM, a multimodal vision-language foundation model (VLM) designed to unify broad diagnostic capability with fine-grain explainability. The development and testing of EVLF-FM encompassed over 1.3 million total samples from 23 global datasets across eleven imaging modalities related to six clinical specialties: dermatology, hepatology, ophthalmology, pathology, pulmonology, and radiology. External validation employed 8,884 independent test samples from 10 additional datasets across five imaging modalities. Technically, EVLF-FM is developed to assist with multiple disease diagnosis and visual question answering with pixel-level visual grounding and reasoning capabilities. In internal validation for disease diagnostics, EVLF-FM achieved the highest average accuracy (0.858) and F1-score (0.797), outperforming leading generalist and specialist models. In medical visual grounding, EVLF-FM also achieved stellar performance across nine modalities with average mIOU of 0.743 and Acc@0.5 of 0.837. External validations further confirmed strong zero-shot and few-shot performance, with competitive F1-scores despite a smaller model size. Through a hybrid training strategy combining supervised and visual reinforcement fine-tuning, EVLF-FM not only achieves state-of-the-art accuracy but also exhibits step-by-step reasoning, aligning outputs with visual evidence. EVLF-FM is an early multi-disease VLM model with explainability and reasoning capabilities that could advance adoption of and trust in foundation models for real-world clinical deployment. |
| title | EVLF-FM: Explainable Vision Language Foundation Model for Medicine |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.24231 |