Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917120754843648 |
|---|---|
| author | Chen, Tianle Chakka, Chaitanya Akula, Arjun Reddy Thomas, Xavier Ghadiyaram, Deepti |
| author_facet | Chen, Tianle Chakka, Chaitanya Akula, Arjun Reddy Thomas, Xavier Ghadiyaram, Deepti |
| contents | Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning. Code and dataset will be publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_22826 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs Chen, Tianle Chakka, Chaitanya Akula, Arjun Reddy Thomas, Xavier Ghadiyaram, Deepti Computer Vision and Pattern Recognition Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning. Code and dataset will be publicly available. |
| title | Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.22826 |