Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910528685735936 |
|---|---|
| author | Bhagwatkar, Rishika Nayak, Shravan Bayat, Reza Roger, Alexis Kaplan, Daniel Z Bashivan, Pouya Rish, Irina |
| author_facet | Bhagwatkar, Rishika Nayak, Shravan Bayat, Reza Roger, Alexis Kaplan, Daniel Z Bashivan, Pouya Rish, Irina |
| contents | Vision-Language Models (VLMs) have witnessed a surge in both research and real-world applications. However, as they are becoming increasingly prevalent, ensuring their robustness against adversarial attacks is paramount. This work systematically investigates the impact of model design choices on the adversarial robustness of VLMs against image-based attacks. Additionally, we introduce novel, cost-effective approaches to enhance robustness through prompt formatting. By rephrasing questions and suggesting potential adversarial perturbations, we demonstrate substantial improvements in model robustness against strong image-based attacks such as Auto-PGD. Our findings provide important guidelines for developing more robust VLMs, particularly for deployment in safety-critical environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_11121 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques Bhagwatkar, Rishika Nayak, Shravan Bayat, Reza Roger, Alexis Kaplan, Daniel Z Bashivan, Pouya Rish, Irina Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Vision-Language Models (VLMs) have witnessed a surge in both research and real-world applications. However, as they are becoming increasingly prevalent, ensuring their robustness against adversarial attacks is paramount. This work systematically investigates the impact of model design choices on the adversarial robustness of VLMs against image-based attacks. Additionally, we introduce novel, cost-effective approaches to enhance robustness through prompt formatting. By rephrasing questions and suggesting potential adversarial perturbations, we demonstrate substantial improvements in model robustness against strong image-based attacks such as Auto-PGD. Our findings provide important guidelines for developing more robust VLMs, particularly for deployment in safety-critical environments. |
| title | Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2407.11121 |