Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2501.15798 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916912577904640 |
|---|---|
| author | Wu, Ruiqi Su, Na Zhang, Chenran Ma, Tengfei Zhou, Tao Cui, Zhiting Tang, Nianfeng Mao, Tianyu Zhou, Yi Fan, Wen Wu, Tianxing Jing, Shenqi Fu, Huazhu |
| author_facet | Wu, Ruiqi Su, Na Zhang, Chenran Ma, Tengfei Zhou, Tao Cui, Zhiting Tang, Nianfeng Mao, Tianyu Zhou, Yi Fan, Wen Wu, Tianxing Jing, Shenqi Fu, Huazhu |
| contents | Vision-language pretraining (VLP) has been investigated to generalize across diverse downstream tasks for fundus image analysis. Although recent methods showcase promising achievements, they significantly rely on large-scale private image-text data but pay less attention to the pretraining manner, which limits their further advancements. In this work, we introduce MM-Retinal V2, a high-quality image-text paired dataset comprising CFP, FFA, and OCT image modalities. Then, we propose a novel fundus vision-language pretraining model, namely KeepFIT V2, which is pretrained by integrating knowledge from the elite data spark into categorical public datasets. Specifically, a preliminary textual pretraining is adopted to equip the text encoder with primarily ophthalmic textual knowledge. Moreover, a hybrid image-text knowledge injection module is designed for knowledge transfer, which is essentially based on a combination of global semantic concepts from contrastive learning and local appearance details from generative learning. Extensive experiments across zero-shot, few-shot, and linear probing settings highlight the generalization and transferability of KeepFIT V2, delivering performance competitive to state-of-the-art fundus VLP models trained on large-scale private image-text datasets. Our dataset and model are publicly available via https://github.com/lxirich/MM-Retinal. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15798 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining Wu, Ruiqi Su, Na Zhang, Chenran Ma, Tengfei Zhou, Tao Cui, Zhiting Tang, Nianfeng Mao, Tianyu Zhou, Yi Fan, Wen Wu, Tianxing Jing, Shenqi Fu, Huazhu Computer Vision and Pattern Recognition Vision-language pretraining (VLP) has been investigated to generalize across diverse downstream tasks for fundus image analysis. Although recent methods showcase promising achievements, they significantly rely on large-scale private image-text data but pay less attention to the pretraining manner, which limits their further advancements. In this work, we introduce MM-Retinal V2, a high-quality image-text paired dataset comprising CFP, FFA, and OCT image modalities. Then, we propose a novel fundus vision-language pretraining model, namely KeepFIT V2, which is pretrained by integrating knowledge from the elite data spark into categorical public datasets. Specifically, a preliminary textual pretraining is adopted to equip the text encoder with primarily ophthalmic textual knowledge. Moreover, a hybrid image-text knowledge injection module is designed for knowledge transfer, which is essentially based on a combination of global semantic concepts from contrastive learning and local appearance details from generative learning. Extensive experiments across zero-shot, few-shot, and linear probing settings highlight the generalization and transferability of KeepFIT V2, delivering performance competitive to state-of-the-art fundus VLP models trained on large-scale private image-text datasets. Our dataset and model are publicly available via https://github.com/lxirich/MM-Retinal. |
| title | MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.15798 |