CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910477519421440 |
|---|---|
| author | Javed, Sajid Mahmood, Arif Ganapathi, Iyyakutti Iyappan Dharejo, Fayaz Ali Werghi, Naoufel Bennamoun, Mohammed |
| author_facet | Javed, Sajid Mahmood, Arif Ganapathi, Iyyakutti Iyappan Dharejo, Fayaz Ali Werghi, Naoufel Bennamoun, Mohammed |
| contents | This paper proposes Comprehensive Pathology Language Image Pre-training (CPLIP), a new unsupervised technique designed to enhance the alignment of images and text in histopathology for tasks such as classification and segmentation. This methodology enriches vision-language models by leveraging extensive data without needing ground truth annotations. CPLIP involves constructing a pathology-specific dictionary, generating textual descriptions for images using language models, and retrieving relevant images for each text snippet via a pre-trained model. The model is then fine-tuned using a many-to-many contrastive learning method to align complex interrelated concepts across both modalities. Evaluated across multiple histopathology tasks, CPLIP shows notable improvements in zero-shot learning scenarios, outperforming existing methods in both interpretability and robustness and setting a higher benchmark for the application of vision-language models in the field. To encourage further research and replication, the code for CPLIP is available on GitHub at https://cplip.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05205 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment Javed, Sajid Mahmood, Arif Ganapathi, Iyyakutti Iyappan Dharejo, Fayaz Ali Werghi, Naoufel Bennamoun, Mohammed Computer Vision and Pattern Recognition Computation and Language Machine Learning Multimedia Image and Video Processing This paper proposes Comprehensive Pathology Language Image Pre-training (CPLIP), a new unsupervised technique designed to enhance the alignment of images and text in histopathology for tasks such as classification and segmentation. This methodology enriches vision-language models by leveraging extensive data without needing ground truth annotations. CPLIP involves constructing a pathology-specific dictionary, generating textual descriptions for images using language models, and retrieving relevant images for each text snippet via a pre-trained model. The model is then fine-tuned using a many-to-many contrastive learning method to align complex interrelated concepts across both modalities. Evaluated across multiple histopathology tasks, CPLIP shows notable improvements in zero-shot learning scenarios, outperforming existing methods in both interpretability and robustness and setting a higher benchmark for the application of vision-language models in the field. To encourage further research and replication, the code for CPLIP is available on GitHub at https://cplip.github.io/ |
| title | CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning Multimedia Image and Video Processing |
| url | https://arxiv.org/abs/2406.05205 |