Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.08541 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908949807104000 |
|---|---|
| author | Xu, Haolei Hong, Haiwen Li, Hongxing Zhou, Rui Zhang, Yang Huang, Longtao Xue, Hui Shen, Yongliang Lu, Weiming Zhuang, Yueting |
| author_facet | Xu, Haolei Hong, Haiwen Li, Hongxing Zhou, Rui Zhang, Yang Huang, Longtao Xue, Hui Shen, Yongliang Lu, Weiming Zhuang, Yueting |
| contents | Multimodal Mixture-of-Experts (MoE) models have achieved remarkable performance on vision-language tasks. However, we identify a puzzling phenomenon termed Seeing but Not Thinking: models accurately perceive image content yet fail in subsequent reasoning, while correctly solving identical problems presented as pure text. Through systematic analysis, we first verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation. We then reveal that visual experts and domain experts exhibit layer-wise separation, with image inputs inducing significant routing divergence from text inputs in middle layers where domain experts concentrate. Based on these findings, we propose the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts. To validate this hypothesis, we design a routing-guided intervention method that enhances domain expert activation. Experiments on three multimodal MoE models across six benchmarks demonstrate consistent improvements, with gains of up to 3.17% on complex visual reasoning tasks. Our analysis further reveals that domain expert identification locates cognitive functions rather than sample-specific solutions, enabling effective transfer across tasks with different information structures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08541 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts Xu, Haolei Hong, Haiwen Li, Hongxing Zhou, Rui Zhang, Yang Huang, Longtao Xue, Hui Shen, Yongliang Lu, Weiming Zhuang, Yueting Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimodal Mixture-of-Experts (MoE) models have achieved remarkable performance on vision-language tasks. However, we identify a puzzling phenomenon termed Seeing but Not Thinking: models accurately perceive image content yet fail in subsequent reasoning, while correctly solving identical problems presented as pure text. Through systematic analysis, we first verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation. We then reveal that visual experts and domain experts exhibit layer-wise separation, with image inputs inducing significant routing divergence from text inputs in middle layers where domain experts concentrate. Based on these findings, we propose the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts. To validate this hypothesis, we design a routing-guided intervention method that enhances domain expert activation. Experiments on three multimodal MoE models across six benchmarks demonstrate consistent improvements, with gains of up to 3.17% on complex visual reasoning tasks. Our analysis further reveals that domain expert identification locates cognitive functions rather than sample-specific solutions, enabling effective transfer across tasks with different information structures. |
| title | Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2604.08541 |