Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.08668 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909887790841856 |
|---|---|
| author | Jiang, Songtao Wang, Yuan Song, Sibo Hu, Tianxiang Zhou, Chenyi Pu, Bin Zhang, Yan Yang, Zhibo Feng, Yang Zhou, Joey Tianyi Hao, Jin Chen, Zijian Wu, Ruijia Tang, Tao Lv, Junhui Xu, Hongxia Wang, Hongwei Xiao, Jun Feng, Bin Zhu, Fudong Li, Kenli Xie, Weidi Sun, Jimeng Wu, Jian Liu, Zuozhu |
| author_facet | Jiang, Songtao Wang, Yuan Song, Sibo Hu, Tianxiang Zhou, Chenyi Pu, Bin Zhang, Yan Yang, Zhibo Feng, Yang Zhou, Joey Tianyi Hao, Jin Chen, Zijian Wu, Ruijia Tang, Tao Lv, Junhui Xu, Hongxia Wang, Hongwei Xiao, Jun Feng, Bin Zhu, Fudong Li, Kenli Xie, Weidi Sun, Jimeng Wu, Jian Liu, Zuozhu |
| contents | Real-world clinical decision-making requires integrating heterogeneous data, including medical text, 2D images, 3D volumes, and videos, while existing AI systems fail to unify all these signals, limiting their utility. In this paper, we introduce Hulu-Med, a transparent, generalist medical Vision-Language Model (VLM) designed to unify language-only, 2D/3D vision-language, and video understanding within a single architecture. Hulu-Med is trained on a curated corpus of 16.7 million samples, comprising exclusively public or synthetic data, spanning 12 major anatomical systems and 14 medical imaging modalities. Hulu-Med employs a medical-aware token-reduction strategy that prunes redundant visual tokens, achieving up to a 55% reduction for 3D and video inputs, improving cross-modal efficiency, and enabling training at 7B-32B parameter scales in approximately 4,000-40,000 GPU hours. Across 30 public in-domain and out-of-domain medical benchmarks-covering text reasoning, visual question answering, report generation, multilingual dialogue, video understanding, and rare disease diagnosis-Hulu-Med surpasses existing open-source models on 27 of 30 benchmarks and outperforms proprietary systems such as GPT-4o on 16 benchmarks. Despite being a VLM, Hulu-Med outperforms GPT-4o and matches GPT-o1 on the text-only HealthBench. For the first time in the community, we provide a fully transparent, reproducible and cost-effective pipeline for holistic medical vision-language understanding by releasing our end-to-end data curation, training procedures, and model parameters. Code and models are available at https://github.com/ZJUI-AI4H/Hulu-Med. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08668 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding Jiang, Songtao Wang, Yuan Song, Sibo Hu, Tianxiang Zhou, Chenyi Pu, Bin Zhang, Yan Yang, Zhibo Feng, Yang Zhou, Joey Tianyi Hao, Jin Chen, Zijian Wu, Ruijia Tang, Tao Lv, Junhui Xu, Hongxia Wang, Hongwei Xiao, Jun Feng, Bin Zhu, Fudong Li, Kenli Xie, Weidi Sun, Jimeng Wu, Jian Liu, Zuozhu Computer Vision and Pattern Recognition Real-world clinical decision-making requires integrating heterogeneous data, including medical text, 2D images, 3D volumes, and videos, while existing AI systems fail to unify all these signals, limiting their utility. In this paper, we introduce Hulu-Med, a transparent, generalist medical Vision-Language Model (VLM) designed to unify language-only, 2D/3D vision-language, and video understanding within a single architecture. Hulu-Med is trained on a curated corpus of 16.7 million samples, comprising exclusively public or synthetic data, spanning 12 major anatomical systems and 14 medical imaging modalities. Hulu-Med employs a medical-aware token-reduction strategy that prunes redundant visual tokens, achieving up to a 55% reduction for 3D and video inputs, improving cross-modal efficiency, and enabling training at 7B-32B parameter scales in approximately 4,000-40,000 GPU hours. Across 30 public in-domain and out-of-domain medical benchmarks-covering text reasoning, visual question answering, report generation, multilingual dialogue, video understanding, and rare disease diagnosis-Hulu-Med surpasses existing open-source models on 27 of 30 benchmarks and outperforms proprietary systems such as GPT-4o on 16 benchmarks. Despite being a VLM, Hulu-Med outperforms GPT-4o and matches GPT-o1 on the text-only HealthBench. For the first time in the community, we provide a fully transparent, reproducible and cost-effective pipeline for holistic medical vision-language understanding by releasing our end-to-end data curation, training procedures, and model parameters. Code and models are available at https://github.com/ZJUI-AI4H/Hulu-Med. |
| title | Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.08668 |