Toward Inherently Robust VLMs Against Visual Perception Attacks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912886314500096 |
|---|---|
| author | MohajerAnsari, Pedram Salarpour, Amir Kühr, Michael Huang, Siyu Hamad, Mohammad Steinhorst, Sebastian Olufowobi, Habeeb Li, Bing Pesé, Mert D. |
| author_facet | MohajerAnsari, Pedram Salarpour, Amir Kühr, Michael Huang, Siyu Hamad, Mohammad Steinhorst, Sebastian Olufowobi, Habeeb Li, Bing Pesé, Mert D. |
| contents | Autonomous vehicles rely on deep neural networks (DNNs) for traffic sign recognition, lane centering, and vehicle detection, yet these models are vulnerable to attacks that induce misclassification and threaten safety. Existing defenses (e.g., adversarial training) often fail to generalize and degrade clean accuracy. We introduce Vehicle Vision-Language Models (V2LMs), fine-tuned vision-language models specialized for autonomous vehicle perception, and show that they are inherently more robust to unseen attacks without adversarial training, maintaining substantially higher adversarial accuracy than conventional DNNs. We study two deployments: Solo (task-specific V2LMs) and Tandem (a single V2LM for all three tasks). Under attacks, DNNs drop 33-74%, whereas V2LMs decline by under 8% on average. Tandem achieves comparable robustness to Solo while being more memory-efficient. We also explore integrating V2LMs in parallel with existing perception stacks to enhance resilience. Our results suggest V2LMs are a promising path toward secure, robust AV perception. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_11472 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Toward Inherently Robust VLMs Against Visual Perception Attacks MohajerAnsari, Pedram Salarpour, Amir Kühr, Michael Huang, Siyu Hamad, Mohammad Steinhorst, Sebastian Olufowobi, Habeeb Li, Bing Pesé, Mert D. Computer Vision and Pattern Recognition Machine Learning Autonomous vehicles rely on deep neural networks (DNNs) for traffic sign recognition, lane centering, and vehicle detection, yet these models are vulnerable to attacks that induce misclassification and threaten safety. Existing defenses (e.g., adversarial training) often fail to generalize and degrade clean accuracy. We introduce Vehicle Vision-Language Models (V2LMs), fine-tuned vision-language models specialized for autonomous vehicle perception, and show that they are inherently more robust to unseen attacks without adversarial training, maintaining substantially higher adversarial accuracy than conventional DNNs. We study two deployments: Solo (task-specific V2LMs) and Tandem (a single V2LM for all three tasks). Under attacks, DNNs drop 33-74%, whereas V2LMs decline by under 8% on average. Tandem achieves comparable robustness to Solo while being more memory-efficient. We also explore integrating V2LMs in parallel with existing perception stacks to enhance resilience. Our results suggest V2LMs are a promising path toward secure, robust AV perception. |
| title | Toward Inherently Robust VLMs Against Visual Perception Attacks |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2506.11472 |