Backdooring Vision-Language Models with Out-Of-Distribution Data
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909515972083712 |
|---|---|
| author | Lyu, Weimin Yao, Jiachen Gupta, Saumya Pang, Lu Sun, Tao Yi, Lingjie Hu, Lijie Ling, Haibin Chen, Chao |
| author_facet | Lyu, Weimin Yao, Jiachen Gupta, Saumya Pang, Lu Sun, Tao Yi, Lingjie Hu, Lijie Ling, Haibin Chen, Chao |
| contents | The emergence of Vision-Language Models (VLMs) represents a significant advancement in integrating computer vision with Large Language Models (LLMs) to generate detailed text descriptions from visual inputs. Despite their growing importance, the security of VLMs, particularly against backdoor attacks, is under explored. Moreover, prior works often assume attackers have access to the original training data, which is often unrealistic. In this paper, we address a more practical and challenging scenario where attackers must rely solely on Out-Of-Distribution (OOD) data. We introduce VLOOD (Backdooring Vision-Language Models with Out-of-Distribution Data), a novel approach with two key contributions: (1) demonstrating backdoor attacks on VLMs in complex image-to-text tasks while minimizing degradation of the original semantics under poisoned inputs, and (2) proposing innovative techniques for backdoor injection without requiring any access to the original training data. Our evaluation on image captioning and visual question answering (VQA) tasks confirms the effectiveness of VLOOD, revealing a critical security vulnerability in VLMs and laying the foundation for future research on securing multimodal models against sophisticated threats. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01264 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Backdooring Vision-Language Models with Out-Of-Distribution Data Lyu, Weimin Yao, Jiachen Gupta, Saumya Pang, Lu Sun, Tao Yi, Lingjie Hu, Lijie Ling, Haibin Chen, Chao Computer Vision and Pattern Recognition The emergence of Vision-Language Models (VLMs) represents a significant advancement in integrating computer vision with Large Language Models (LLMs) to generate detailed text descriptions from visual inputs. Despite their growing importance, the security of VLMs, particularly against backdoor attacks, is under explored. Moreover, prior works often assume attackers have access to the original training data, which is often unrealistic. In this paper, we address a more practical and challenging scenario where attackers must rely solely on Out-Of-Distribution (OOD) data. We introduce VLOOD (Backdooring Vision-Language Models with Out-of-Distribution Data), a novel approach with two key contributions: (1) demonstrating backdoor attacks on VLMs in complex image-to-text tasks while minimizing degradation of the original semantics under poisoned inputs, and (2) proposing innovative techniques for backdoor injection without requiring any access to the original training data. Our evaluation on image captioning and visual question answering (VQA) tasks confirms the effectiveness of VLOOD, revealing a critical security vulnerability in VLMs and laying the foundation for future research on securing multimodal models against sophisticated threats. |
| title | Backdooring Vision-Language Models with Out-Of-Distribution Data |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.01264 |