Membership Inference Attacks against Large Vision-Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912105758720000 |
|---|---|
| author | Li, Zhan Wu, Yongtao Chen, Yihang Tonin, Francesco Rocamora, Elias Abad Cevher, Volkan |
| author_facet | Li, Zhan Wu, Yongtao Chen, Yihang Tonin, Francesco Rocamora, Elias Abad Cevher, Volkan |
| contents | Large vision-language models (VLLMs) exhibit promising capabilities for processing multi-modal tasks across various application scenarios. However, their emergence also raises significant data security concerns, given the potential inclusion of sensitive information, such as private photos and medical records, in their training datasets. Detecting inappropriately used data in VLLMs remains a critical and unresolved issue, mainly due to the lack of standardized datasets and suitable methodologies. In this study, we introduce the first membership inference attack (MIA) benchmark tailored for various VLLMs to facilitate training data detection. Then, we propose a novel MIA pipeline specifically designed for token-level image detection. Lastly, we present a new metric called MaxRényi-K%, which is based on the confidence of the model output and applies to both text and image data. We believe that our work can deepen the understanding and methodology of MIAs in the context of VLLMs. Our code and datasets are available at https://github.com/LIONS-EPFL/VL-MIA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_02902 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Membership Inference Attacks against Large Vision-Language Models Li, Zhan Wu, Yongtao Chen, Yihang Tonin, Francesco Rocamora, Elias Abad Cevher, Volkan Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Cryptography and Security Machine Learning Large vision-language models (VLLMs) exhibit promising capabilities for processing multi-modal tasks across various application scenarios. However, their emergence also raises significant data security concerns, given the potential inclusion of sensitive information, such as private photos and medical records, in their training datasets. Detecting inappropriately used data in VLLMs remains a critical and unresolved issue, mainly due to the lack of standardized datasets and suitable methodologies. In this study, we introduce the first membership inference attack (MIA) benchmark tailored for various VLLMs to facilitate training data detection. Then, we propose a novel MIA pipeline specifically designed for token-level image detection. Lastly, we present a new metric called MaxRényi-K%, which is based on the confidence of the model output and applies to both text and image data. We believe that our work can deepen the understanding and methodology of MIAs in the context of VLLMs. Our code and datasets are available at https://github.com/LIONS-EPFL/VL-MIA. |
| title | Membership Inference Attacks against Large Vision-Language Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2411.02902 |