MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909060142465024 |
|---|---|
| author | Liu, Jiarun Zhou, Hong-Yu Li, Cheng Huang, Weijian Yang, Hao Liang, Yong Wang, Shanshan |
| author_facet | Liu, Jiarun Zhou, Hong-Yu Li, Cheng Huang, Weijian Yang, Hao Liang, Yong Wang, Shanshan |
| contents | Existing contrastive language-image pre-training aims to learn a joint representation by matching abundant image-text pairs. However, the number of image-text pairs in medical datasets is usually orders of magnitude smaller than that in natural datasets. Besides, medical image-text pairs often involve numerous complex fine-grained correspondences. This paper aims to enhance the data efficiency by introducing multiple-to-multiple local relationship modeling to capture denser supervisions. More specifically, we propose a Medical Language-Image Pre-training (MLIP) framework, which exploits the limited image-text medical data more efficiently through patch-sentence matching. Furthermore, we introduce a masked contrastive learning strategy with semantic integrity estimation to reduce redundancy in images while preserving the underlying semantics. Our evaluation results show that MLIP outperforms previous work in zero/few-shot classification and few-shot segmentation tasks by a large margin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_01591 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning Liu, Jiarun Zhou, Hong-Yu Li, Cheng Huang, Weijian Yang, Hao Liang, Yong Wang, Shanshan Computer Vision and Pattern Recognition Existing contrastive language-image pre-training aims to learn a joint representation by matching abundant image-text pairs. However, the number of image-text pairs in medical datasets is usually orders of magnitude smaller than that in natural datasets. Besides, medical image-text pairs often involve numerous complex fine-grained correspondences. This paper aims to enhance the data efficiency by introducing multiple-to-multiple local relationship modeling to capture denser supervisions. More specifically, we propose a Medical Language-Image Pre-training (MLIP) framework, which exploits the limited image-text medical data more efficiently through patch-sentence matching. Furthermore, we introduce a masked contrastive learning strategy with semantic integrity estimation to reduce redundancy in images while preserving the underlying semantics. Our evaluation results show that MLIP outperforms previous work in zero/few-shot classification and few-shot segmentation tasks by a large margin. |
| title | MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.01591 |