Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911395665149952 |
|---|---|
| author | Liu, Zhining Wang, Tianyi Lin, Xiao Ouyang, Penghao Li, Gaotang Yang, Ze Liu, Hui Keswani, Sumit Pardeshi, Vishwa Zhao, Huijun Fan, Wei Tong, Hanghang |
| author_facet | Liu, Zhining Wang, Tianyi Lin, Xiao Ouyang, Penghao Li, Gaotang Yang, Ze Liu, Hui Keswani, Sumit Pardeshi, Vishwa Zhao, Huijun Fan, Wei Tong, Hanghang |
| contents | Despite substantial efforts toward improving the moral alignment of Vision-Language Models (VLMs), it remains unclear whether their ethical judgments are stable in realistic settings. This work studies moral robustness in VLMs, defined as the ability to preserve moral judgments under textual and visual perturbations that do not alter the underlying moral context. We systematically probe VLMs with a diverse set of model-agnostic multimodal perturbations and find that their moral stances are highly fragile, frequently flipping under simple manipulations. Our analysis reveals systematic vulnerabilities across perturbation types, moral domains, and model scales, including a sycophancy trade-off where stronger instruction-following models are more susceptible to persuasion. We further show that lightweight inference-time interventions can partially restore moral stability. These results demonstrate that moral alignment alone is insufficient and that moral robustness is a necessary criterion for the responsible deployment of VLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17082 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models Liu, Zhining Wang, Tianyi Lin, Xiao Ouyang, Penghao Li, Gaotang Yang, Ze Liu, Hui Keswani, Sumit Pardeshi, Vishwa Zhao, Huijun Fan, Wei Tong, Hanghang Computers and Society Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Despite substantial efforts toward improving the moral alignment of Vision-Language Models (VLMs), it remains unclear whether their ethical judgments are stable in realistic settings. This work studies moral robustness in VLMs, defined as the ability to preserve moral judgments under textual and visual perturbations that do not alter the underlying moral context. We systematically probe VLMs with a diverse set of model-agnostic multimodal perturbations and find that their moral stances are highly fragile, frequently flipping under simple manipulations. Our analysis reveals systematic vulnerabilities across perturbation types, moral domains, and model scales, including a sycophancy trade-off where stronger instruction-following models are more susceptible to persuasion. We further show that lightweight inference-time interventions can partially restore moral stability. These results demonstrate that moral alignment alone is insufficient and that moral robustness is a necessary criterion for the responsible deployment of VLMs. |
| title | Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models |
| topic | Computers and Society Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2601.17082 |