Steering Multimodal Large Language Models Decoding for Context-Aware Safety
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909802606624768 |
|---|---|
| author | Liu, Zheyuan Xu, Zhangchen Dou, Guangyao Yuan, Xiangchi Tan, Zhaoxuan Poovendran, Radha Jiang, Meng |
| author_facet | Liu, Zheyuan Xu, Zhangchen Dou, Guangyao Yuan, Xiangchi Tan, Zhaoxuan Poovendran, Radha Jiang, Meng |
| contents | Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisions remains limited. Existing methods often fail to balance oversensitivity (unjustified refusals of benign queries) and undersensitivity (missed detection of visually grounded risks), leaving a persistent gap in safety alignment. To address this issue, we introduce Safety-aware Contrastive Decoding (SafeCoDe), a lightweight and model-agnostic decoding framework that dynamically adjusts token generation based on multimodal context. SafeCoDe operates in two stages: (1) a contrastive decoding mechanism that highlights tokens sensitive to visual context by contrasting real and Gaussian-noised images, and (2) a global-aware token modulation strategy that integrates scene-level reasoning with token-level adjustment to adapt refusals according to the predicted safety verdict. Extensive experiments across diverse MLLM architectures and safety benchmarks, covering undersensitivity, oversensitivity, and general safety evaluations, show that SafeCoDe consistently improves context-sensitive refusal behaviors while preserving model helpfulness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19212 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Steering Multimodal Large Language Models Decoding for Context-Aware Safety Liu, Zheyuan Xu, Zhangchen Dou, Guangyao Yuan, Xiangchi Tan, Zhaoxuan Poovendran, Radha Jiang, Meng Computation and Language Artificial Intelligence Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisions remains limited. Existing methods often fail to balance oversensitivity (unjustified refusals of benign queries) and undersensitivity (missed detection of visually grounded risks), leaving a persistent gap in safety alignment. To address this issue, we introduce Safety-aware Contrastive Decoding (SafeCoDe), a lightweight and model-agnostic decoding framework that dynamically adjusts token generation based on multimodal context. SafeCoDe operates in two stages: (1) a contrastive decoding mechanism that highlights tokens sensitive to visual context by contrasting real and Gaussian-noised images, and (2) a global-aware token modulation strategy that integrates scene-level reasoning with token-level adjustment to adapt refusals according to the predicted safety verdict. Extensive experiments across diverse MLLM architectures and safety benchmarks, covering undersensitivity, oversensitivity, and general safety evaluations, show that SafeCoDe consistently improves context-sensitive refusal behaviors while preserving model helpfulness. |
| title | Steering Multimodal Large Language Models Decoding for Context-Aware Safety |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.19212 |