When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915597471711232 |
|---|---|
| author | Zhang, Chenyu Kim, Minsol Ghorbani, Shohreh Wu, Jingyao Picard, Rosalind Maes, Patricia Liang, Paul Pu |
| author_facet | Zhang, Chenyu Kim, Minsol Ghorbani, Shohreh Wu, Jingyao Picard, Rosalind Maes, Patricia Liang, Paul Pu |
| contents | Despite rapid growth in multimodal large language models (MLLMs), their reasoning traces remain opaque: it is often unclear which modality drives a prediction, how conflicts are resolved, or when one stream dominates. In this paper, we introduce modality sabotage, a diagnostic failure mode in which a high-confidence unimodal error overrides other evidence and misleads the fused result. To analyze such dynamics, we propose a lightweight, model-agnostic evaluation layer that treats each modality as an agent, producing candidate labels and a brief self-assessment used for auditing. A simple fusion mechanism aggregates these outputs, exposing contributors (modalities supporting correct outcomes) and saboteurs (modalities that mislead). Applying our diagnostic layer in a case study on multimodal emotion recognition benchmarks with foundation models revealed systematic reliability profiles, providing insight into whether failures may arise from dataset artifacts or model limitations. More broadly, our framework offers a diagnostic scaffold for multimodal reasoning, supporting principled auditing of fusion dynamics and informing possible interventions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_02794 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning Zhang, Chenyu Kim, Minsol Ghorbani, Shohreh Wu, Jingyao Picard, Rosalind Maes, Patricia Liang, Paul Pu Artificial Intelligence Multiagent Systems Despite rapid growth in multimodal large language models (MLLMs), their reasoning traces remain opaque: it is often unclear which modality drives a prediction, how conflicts are resolved, or when one stream dominates. In this paper, we introduce modality sabotage, a diagnostic failure mode in which a high-confidence unimodal error overrides other evidence and misleads the fused result. To analyze such dynamics, we propose a lightweight, model-agnostic evaluation layer that treats each modality as an agent, producing candidate labels and a brief self-assessment used for auditing. A simple fusion mechanism aggregates these outputs, exposing contributors (modalities supporting correct outcomes) and saboteurs (modalities that mislead). Applying our diagnostic layer in a case study on multimodal emotion recognition benchmarks with foundation models revealed systematic reliability profiles, providing insight into whether failures may arise from dataset artifacts or model limitations. More broadly, our framework offers a diagnostic scaffold for multimodal reasoning, supporting principled auditing of fusion dynamics and informing possible interventions. |
| title | When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning |
| topic | Artificial Intelligence Multiagent Systems |
| url | https://arxiv.org/abs/2511.02794 |