A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914321917804544 |
|---|---|
| author | Yan, Siyuan Li, Xieji Mo, Dan Tschandl, Philipp Jiang, Yiwen Wang, Zhonghua Hu, Ming Ju, Lie Vico-Alonso, Cristina Zheng, Yizhen Liu, Jiahe Zhou, Juexiao Chello, Camilla Cheung, Jen G. Anriot, Julien Thomas, Luc Primiero, Clare Tan, Gin Ng, Aik Beng See, Simon Tang, Xiaoying Ip, Albert Liao, Xiaoyang Bowling, Adrian Haskett, Martin Zhao, Shuang Janda, Monika Soyer, H. Peter Mar, Victoria Kittler, Harald Ge, Zongyuan |
| author_facet | Yan, Siyuan Li, Xieji Mo, Dan Tschandl, Philipp Jiang, Yiwen Wang, Zhonghua Hu, Ming Ju, Lie Vico-Alonso, Cristina Zheng, Yizhen Liu, Jiahe Zhou, Juexiao Chello, Camilla Cheung, Jen G. Anriot, Julien Thomas, Luc Primiero, Clare Tan, Gin Ng, Aik Beng See, Simon Tang, Xiaoying Ip, Albert Liao, Xiaoyang Bowling, Adrian Haskett, Martin Zhao, Shuang Janda, Monika Soyer, H. Peter Mar, Victoria Kittler, Harald Ge, Zongyuan |
| contents | Medical foundation models have shown promise in controlled benchmarks, yet widespread deployment remains hindered by reliance on task-specific fine-tuning. Here, we introduce DermFM-Zero, a dermatology vision-language foundation model trained via masked latent modelling and contrastive learning on over 4 million multimodal data points. We evaluated DermFM-Zero across 20 benchmarks spanning zero-shot diagnosis and multimodal retrieval, achieving state-of-the-art performance without task-specific adaptation. We further evaluated its zero-shot capabilities in three multinational reader studies involving over 1,100 clinicians. In primary care settings, AI assistance enabled general practitioners to nearly double their differential diagnostic accuracy across 98 skin conditions. In specialist settings, the model significantly outperformed board-certified dermatologists in multimodal skin cancer assessment. In collaborative workflows, AI assistance enabled non-experts to surpass unassisted experts while improving management appropriateness. Finally, we show that DermFM-Zero's latent representations are interpretable: sparse autoencoders unsupervisedly disentangle clinically meaningful concepts that outperform predefined-vocabulary approaches and enable targeted suppression of artifact-induced biases, enhancing robustness without retraining. These findings demonstrate that a foundation model can provide effective, safe, and transparent zero-shot clinical decision support. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_10624 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology Yan, Siyuan Li, Xieji Mo, Dan Tschandl, Philipp Jiang, Yiwen Wang, Zhonghua Hu, Ming Ju, Lie Vico-Alonso, Cristina Zheng, Yizhen Liu, Jiahe Zhou, Juexiao Chello, Camilla Cheung, Jen G. Anriot, Julien Thomas, Luc Primiero, Clare Tan, Gin Ng, Aik Beng See, Simon Tang, Xiaoying Ip, Albert Liao, Xiaoyang Bowling, Adrian Haskett, Martin Zhao, Shuang Janda, Monika Soyer, H. Peter Mar, Victoria Kittler, Harald Ge, Zongyuan Computer Vision and Pattern Recognition Artificial Intelligence Medical foundation models have shown promise in controlled benchmarks, yet widespread deployment remains hindered by reliance on task-specific fine-tuning. Here, we introduce DermFM-Zero, a dermatology vision-language foundation model trained via masked latent modelling and contrastive learning on over 4 million multimodal data points. We evaluated DermFM-Zero across 20 benchmarks spanning zero-shot diagnosis and multimodal retrieval, achieving state-of-the-art performance without task-specific adaptation. We further evaluated its zero-shot capabilities in three multinational reader studies involving over 1,100 clinicians. In primary care settings, AI assistance enabled general practitioners to nearly double their differential diagnostic accuracy across 98 skin conditions. In specialist settings, the model significantly outperformed board-certified dermatologists in multimodal skin cancer assessment. In collaborative workflows, AI assistance enabled non-experts to surpass unassisted experts while improving management appropriateness. Finally, we show that DermFM-Zero's latent representations are interpretable: sparse autoencoders unsupervisedly disentangle clinically meaningful concepts that outperform predefined-vocabulary approaches and enable targeted suppression of artifact-induced biases, enhancing robustness without retraining. These findings demonstrate that a foundation model can provide effective, safe, and transparent zero-shot clinical decision support. |
| title | A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2602.10624 |