Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912601480364032 |
|---|---|
| author | Zhang, Tong Gao, Kuofeng Bai, Jiawang Zhang, Leo Yu Yin, Xin Wang, Zonghui Ji, Shouling Chen, Wenzhi |
| author_facet | Zhang, Tong Gao, Kuofeng Bai, Jiawang Zhang, Leo Yu Yin, Xin Wang, Zonghui Ji, Shouling Chen, Wenzhi |
| contents | Recent studies have shown that Contrastive Language-Image Pre-training (CLIP) models are threatened by targeted data poisoning and backdoor attacks due to massive training image-caption pairs crawled from the Internet. Previous defense methods correct poisoned image-caption pairs by matching a new caption for each image. However, the matching process relies solely on the global representations of images and captions, overlooking fine-grained features of visual and textual features. It may introduce incorrect image-caption pairs and harm the CLIP pre-training. To address their limitations, we propose an Optimal Transport-based framework to reconstruct image-caption pairs, named OTCCLIP. We propose a new optimal transport-based distance measure between fine-grained visual and textual feature sets and re-assign new captions based on the proposed optimal transport distance. Additionally, to further reduce the negative impact of mismatched pairs, we encourage the inter- and intra-modality fine-grained alignment by employing optimal transport-based objective functions. Our experiments demonstrate that OTCCLIP can successfully decrease the attack success rates of poisoning attacks. Also, compared to previous methods, OTCCLIP significantly improves CLIP's zero-shot and linear probing performance trained on poisoned datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_18717 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment Zhang, Tong Gao, Kuofeng Bai, Jiawang Zhang, Leo Yu Yin, Xin Wang, Zonghui Ji, Shouling Chen, Wenzhi Computer Vision and Pattern Recognition Multimedia Recent studies have shown that Contrastive Language-Image Pre-training (CLIP) models are threatened by targeted data poisoning and backdoor attacks due to massive training image-caption pairs crawled from the Internet. Previous defense methods correct poisoned image-caption pairs by matching a new caption for each image. However, the matching process relies solely on the global representations of images and captions, overlooking fine-grained features of visual and textual features. It may introduce incorrect image-caption pairs and harm the CLIP pre-training. To address their limitations, we propose an Optimal Transport-based framework to reconstruct image-caption pairs, named OTCCLIP. We propose a new optimal transport-based distance measure between fine-grained visual and textual feature sets and re-assign new captions based on the proposed optimal transport distance. Additionally, to further reduce the negative impact of mismatched pairs, we encourage the inter- and intra-modality fine-grained alignment by employing optimal transport-based objective functions. Our experiments demonstrate that OTCCLIP can successfully decrease the attack success rates of poisoning attacks. Also, compared to previous methods, OTCCLIP significantly improves CLIP's zero-shot and linear probing performance trained on poisoned datasets. |
| title | Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2509.18717 |