Defeating Cerberus: Concept-Guided Privacy-Leakage Mitigation in Multimodal Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915523094118400 |
|---|---|
| author | Zhang, Boyang Akkus, Istemi Ekin Chen, Ruichuan Dethise, Alice Satzke, Klaus Rimac, Ivica Zhang, Yang |
| author_facet | Zhang, Boyang Akkus, Istemi Ekin Chen, Ruichuan Dethise, Alice Satzke, Klaus Rimac, Ivica Zhang, Yang |
| contents | Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in processing and reasoning over diverse modalities, but their advanced abilities also raise significant privacy concerns, particularly regarding Personally Identifiable Information (PII) leakage. While relevant research has been conducted on single-modal language models to some extent, the vulnerabilities in the multimodal setting have yet to be fully investigated. In this work, we investigate these emerging risks with a focus on vision language models (VLMs), a representative subclass of MLLMs that covers the two modalities most relevant for PII leakage, vision and text. We introduce a concept-guided mitigation approach that identifies and modifies the model's internal states associated with PII-related content. Our method guides VLMs to refuse PII-sensitive tasks effectively and efficiently, without requiring re-training or fine-tuning. We also address the current lack of multimodal PII datasets by constructing various ones that simulate real-world scenarios. Experimental results demonstrate that the method can achieve an average refusal rate of 93.3% for various PII-related tasks with minimal impact on unrelated model performances. We further examine the mitigation's performance under various conditions to show the adaptability of our proposed method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_25525 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Defeating Cerberus: Concept-Guided Privacy-Leakage Mitigation in Multimodal Language Models Zhang, Boyang Akkus, Istemi Ekin Chen, Ruichuan Dethise, Alice Satzke, Klaus Rimac, Ivica Zhang, Yang Cryptography and Security Machine Learning Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in processing and reasoning over diverse modalities, but their advanced abilities also raise significant privacy concerns, particularly regarding Personally Identifiable Information (PII) leakage. While relevant research has been conducted on single-modal language models to some extent, the vulnerabilities in the multimodal setting have yet to be fully investigated. In this work, we investigate these emerging risks with a focus on vision language models (VLMs), a representative subclass of MLLMs that covers the two modalities most relevant for PII leakage, vision and text. We introduce a concept-guided mitigation approach that identifies and modifies the model's internal states associated with PII-related content. Our method guides VLMs to refuse PII-sensitive tasks effectively and efficiently, without requiring re-training or fine-tuning. We also address the current lack of multimodal PII datasets by constructing various ones that simulate real-world scenarios. Experimental results demonstrate that the method can achieve an average refusal rate of 93.3% for various PII-related tasks with minimal impact on unrelated model performances. We further examine the mitigation's performance under various conditions to show the adaptability of our proposed method. |
| title | Defeating Cerberus: Concept-Guided Privacy-Leakage Mitigation in Multimodal Language Models |
| topic | Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2509.25525 |