Generalization Bounds for Robust Contrastive Learning: From Theory to Practice
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908612973035520 |
|---|---|
| author | Tran, Ngoc N. Tran, Lam Phan, Hoang Bui, Anh Pham, Tung Tran, Toan Phung, Dinh Le, Trung |
| author_facet | Tran, Ngoc N. Tran, Lam Phan, Hoang Bui, Anh Pham, Tung Tran, Toan Phung, Dinh Le, Trung |
| contents | Contrastive Learning first extracts features from unlabeled data, followed by linear probing with labeled data. Adversarial Contrastive Learning (ACL) integrates Adversarial Training into the first phase to enhance feature robustness against attacks in the probing phase. While ACL has shown strong empirical results, its theoretical understanding remains limited. Furthermore, while a fair amount of theoretical works analyze how the unsupervised loss can support the supervised loss in the probing phase, none has examined its role to the robust supervised loss. To fill this gap, our work develops rigorous theories to identify which components in the unsupervised training can help improve the robust supervised loss. Specifically, besides the adversarial contrastive loss, we reveal that the benign one, along with a global divergence between benign and adversarial examples can also improve robustness. Proper experiments are conducted to justify our findings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09671 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Generalization Bounds for Robust Contrastive Learning: From Theory to Practice Tran, Ngoc N. Tran, Lam Phan, Hoang Bui, Anh Pham, Tung Tran, Toan Phung, Dinh Le, Trung Machine Learning Computer Vision and Pattern Recognition Contrastive Learning first extracts features from unlabeled data, followed by linear probing with labeled data. Adversarial Contrastive Learning (ACL) integrates Adversarial Training into the first phase to enhance feature robustness against attacks in the probing phase. While ACL has shown strong empirical results, its theoretical understanding remains limited. Furthermore, while a fair amount of theoretical works analyze how the unsupervised loss can support the supervised loss in the probing phase, none has examined its role to the robust supervised loss. To fill this gap, our work develops rigorous theories to identify which components in the unsupervised training can help improve the robust supervised loss. Specifically, besides the adversarial contrastive loss, we reveal that the benign one, along with a global divergence between benign and adversarial examples can also improve robustness. Proper experiments are conducted to justify our findings. |
| title | Generalization Bounds for Robust Contrastive Learning: From Theory to Practice |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.09671 |