LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914174236360704 |
|---|---|
| author | Khodabandeh, Borna Afzali, Amirabbas Afsharrad, Amirhossein Mousavi, Seyed Shahabeddin Lall, Sanjay Amini, Sajjad Moosavi-Dezfooli, Seyed-Mohsen |
| author_facet | Khodabandeh, Borna Afzali, Amirabbas Afsharrad, Amirhossein Mousavi, Seyed Shahabeddin Lall, Sanjay Amini, Sajjad Moosavi-Dezfooli, Seyed-Mohsen |
| contents | Visual encoders have become fundamental components in modern computer vision pipelines. However, ensuring robustness against adversarial perturbations remains a critical challenge. Recent efforts have explored both supervised and unsupervised adversarial fine-tuning strategies. We identify two key limitations in these approaches: (i) they often suffer from instability, especially during the early stages of fine-tuning, resulting in suboptimal convergence and degraded performance on clean data, and (ii) they exhibit a suboptimal trade-off between robustness and clean data accuracy, hindering the simultaneous optimization of both objectives. To overcome these challenges, we propose Lagrangian-Optimized Robust Embeddings (LORE), a novel unsupervised adversarial fine-tuning framework. LORE utilizes constrained optimization, which offers a principled approach to balancing competing goals, such as improving robustness while preserving nominal performance. By enforcing embedding-space proximity constraints, LORE effectively maintains clean data performance throughout adversarial fine-tuning. Extensive experiments show that LORE significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy. Furthermore, we demonstrate the effectiveness of the adversarially fine-tuned CLIP image encoder in out-of-distribution generalization and enhancing the interpretability of image embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18884 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders Khodabandeh, Borna Afzali, Amirabbas Afsharrad, Amirhossein Mousavi, Seyed Shahabeddin Lall, Sanjay Amini, Sajjad Moosavi-Dezfooli, Seyed-Mohsen Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Optimization and Control Visual encoders have become fundamental components in modern computer vision pipelines. However, ensuring robustness against adversarial perturbations remains a critical challenge. Recent efforts have explored both supervised and unsupervised adversarial fine-tuning strategies. We identify two key limitations in these approaches: (i) they often suffer from instability, especially during the early stages of fine-tuning, resulting in suboptimal convergence and degraded performance on clean data, and (ii) they exhibit a suboptimal trade-off between robustness and clean data accuracy, hindering the simultaneous optimization of both objectives. To overcome these challenges, we propose Lagrangian-Optimized Robust Embeddings (LORE), a novel unsupervised adversarial fine-tuning framework. LORE utilizes constrained optimization, which offers a principled approach to balancing competing goals, such as improving robustness while preserving nominal performance. By enforcing embedding-space proximity constraints, LORE effectively maintains clean data performance throughout adversarial fine-tuning. Extensive experiments show that LORE significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy. Furthermore, we demonstrate the effectiveness of the adversarially fine-tuned CLIP image encoder in out-of-distribution generalization and enhancing the interpretability of image embeddings. |
| title | LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Optimization and Control |
| url | https://arxiv.org/abs/2505.18884 |