A multi-modal vision-language model for generalizable annotation-free pathology localization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909893558009856 |
|---|---|
| author | Yang, Hao Zhou, Hong-Yu Liu, Jiarun Huang, Weijian Li, Cheng Li, Zhihuan Gao, Yuanxu Liu, Qiegen Liang, Yong Yang, Qi Wu, Song Tan, Tao Zheng, Hairong Zhang, Kang Wang, Shanshan |
| author_facet | Yang, Hao Zhou, Hong-Yu Liu, Jiarun Huang, Weijian Li, Cheng Li, Zhihuan Gao, Yuanxu Liu, Qiegen Liang, Yong Yang, Qi Wu, Song Tan, Tao Zheng, Hairong Zhang, Kang Wang, Shanshan |
| contents | Existing deep learning models for defining pathology from clinical imaging data rely on expert annotations and lack generalization capabilities in open clinical environments. Here, we present a generalizable vision-language model for Annotation-Free pathology Localization (AFLoc). The core strength of AFLoc is extensive multi-level semantic structure-based contrastive learning, which comprehensively aligns multi-granularity medical concepts with abundant image features to adapt to the diverse expressions of pathologies without the reliance on expert image annotations. We conduct primary experiments on a dataset of 220K pairs of image-report chest X-ray images and perform validation across eight external datasets encompassing 34 types of chest pathologies. The results demonstrate that AFLoc outperforms state-of-the-art methods in both annotation-free localization and classification tasks. Additionally, we assess the generalizability of AFLoc on other modalities, including histopathology and retinal fundus images. We show that AFLoc exhibits robust generalization capabilities, even surpassing human benchmarks in localizing five different types of pathological images. These results highlight the potential of AFLoc in reducing annotation requirements and its applicability in complex clinical environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02044 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A multi-modal vision-language model for generalizable annotation-free pathology localization Yang, Hao Zhou, Hong-Yu Liu, Jiarun Huang, Weijian Li, Cheng Li, Zhihuan Gao, Yuanxu Liu, Qiegen Liang, Yong Yang, Qi Wu, Song Tan, Tao Zheng, Hairong Zhang, Kang Wang, Shanshan Computer Vision and Pattern Recognition Existing deep learning models for defining pathology from clinical imaging data rely on expert annotations and lack generalization capabilities in open clinical environments. Here, we present a generalizable vision-language model for Annotation-Free pathology Localization (AFLoc). The core strength of AFLoc is extensive multi-level semantic structure-based contrastive learning, which comprehensively aligns multi-granularity medical concepts with abundant image features to adapt to the diverse expressions of pathologies without the reliance on expert image annotations. We conduct primary experiments on a dataset of 220K pairs of image-report chest X-ray images and perform validation across eight external datasets encompassing 34 types of chest pathologies. The results demonstrate that AFLoc outperforms state-of-the-art methods in both annotation-free localization and classification tasks. Additionally, we assess the generalizability of AFLoc on other modalities, including histopathology and retinal fundus images. We show that AFLoc exhibits robust generalization capabilities, even surpassing human benchmarks in localizing five different types of pathological images. These results highlight the potential of AFLoc in reducing annotation requirements and its applicability in complex clinical environments. |
| title | A multi-modal vision-language model for generalizable annotation-free pathology localization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.02044 |