MuPHI: Learning Implicit Multimodal Harm Reasoning via Semantically Grounded Reward Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916061406822400 |
|---|---|
| author | Saha, Anisha Suresh, Varsha Kamova, Teodora Wiedmann, Sophia Hospedales, Timothy Demberg, Vera |
| author_facet | Saha, Anisha Suresh, Varsha Kamova, Teodora Wiedmann, Sophia Hospedales, Timothy Demberg, Vera |
| contents | Understanding how harm emerges from interaction between otherwise benign image-text pairs requires intent-aware cross-modal reasoning beyond surface-level features. Existing vision-language models (VLMs) excel at literal reasoning over perceptual cues but often fail to derive harmful semantics that rely on implicit, context-dependent reasoning. To evaluate VLMs on compositional harm detection and reasoning, we introduce Multimodal Pragmatic Harm Interpretation (MuPHI), a dataset containing image-text pairs where harm is encoded in subtle multimodal cues. MuPHI spans diverse harm categories and includes annotated harm rationales for assessing VLM reasoning chains. To improve both detection and reasoning in VLMs, we propose MuPHIRM, a reasoning-augmented training framework which learns joint semantics by optimizing multi-perspective rewards. MuPHIRM improves both harm detection and reasoning quality of VLMs while demonstrating superior out-of-distribution robustness compared to both trained and inference-time baselines. Our findings suggest that reasoning-oriented reward optimization offers a promising direction towards building multimodal systems that generalize beyond benchmark-specific shortcuts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_29951 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MuPHI: Learning Implicit Multimodal Harm Reasoning via Semantically Grounded Reward Optimization Saha, Anisha Suresh, Varsha Kamova, Teodora Wiedmann, Sophia Hospedales, Timothy Demberg, Vera Artificial Intelligence Computation and Language Machine Learning Multimedia Understanding how harm emerges from interaction between otherwise benign image-text pairs requires intent-aware cross-modal reasoning beyond surface-level features. Existing vision-language models (VLMs) excel at literal reasoning over perceptual cues but often fail to derive harmful semantics that rely on implicit, context-dependent reasoning. To evaluate VLMs on compositional harm detection and reasoning, we introduce Multimodal Pragmatic Harm Interpretation (MuPHI), a dataset containing image-text pairs where harm is encoded in subtle multimodal cues. MuPHI spans diverse harm categories and includes annotated harm rationales for assessing VLM reasoning chains. To improve both detection and reasoning in VLMs, we propose MuPHIRM, a reasoning-augmented training framework which learns joint semantics by optimizing multi-perspective rewards. MuPHIRM improves both harm detection and reasoning quality of VLMs while demonstrating superior out-of-distribution robustness compared to both trained and inference-time baselines. Our findings suggest that reasoning-oriented reward optimization offers a promising direction towards building multimodal systems that generalize beyond benchmark-specific shortcuts. |
| title | MuPHI: Learning Implicit Multimodal Harm Reasoning via Semantically Grounded Reward Optimization |
| topic | Artificial Intelligence Computation and Language Machine Learning Multimedia |
| url | https://arxiv.org/abs/2605.29951 |