PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915723599675392 |
|---|---|
| author | Wang, Zijing Liu, Yongkang Wang, Mingyang Nie, Ercong Chen, Deyuan Zhao, Zhengjie Feng, Shi Wang, Daling Yang, Xiaocui Zhang, Yifei Schütze, Hinrich |
| author_facet | Wang, Zijing Liu, Yongkang Wang, Mingyang Nie, Ercong Chen, Deyuan Zhao, Zhengjie Feng, Shi Wang, Daling Yang, Xiaocui Zhang, Yifei Schütze, Hinrich |
| contents | Multimodal Large Language Models (MLLMs) rely on strong linguistic reasoning inherited from their base language models. However, multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability, undermining multimodal performance. To address this issue, we propose a training-free framework to mitigate this degradation. Through layer-wise vision token masking, we reveal a common three-stage pattern in multimodal large language models: early-modal separation, mid-modal alignment, and late-modal degradation. By analyzing the behavior of MLLMs at different stages, we propose a plateau-guided model merging method that selectively injects base language model parameters into MLLMs. Experimental results based on five MLLMs on nine benchmarks demonstrate the effectiveness of our method. Attention-based analysis further reveals that merging shifts attention from diffuse, scattered patterns to focused localization on task-relevant visual regions. Our repository is on https://github.com/wzj1718/PlaM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_07645 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs Wang, Zijing Liu, Yongkang Wang, Mingyang Nie, Ercong Chen, Deyuan Zhao, Zhengjie Feng, Shi Wang, Daling Yang, Xiaocui Zhang, Yifei Schütze, Hinrich Computation and Language Multimodal Large Language Models (MLLMs) rely on strong linguistic reasoning inherited from their base language models. However, multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability, undermining multimodal performance. To address this issue, we propose a training-free framework to mitigate this degradation. Through layer-wise vision token masking, we reveal a common three-stage pattern in multimodal large language models: early-modal separation, mid-modal alignment, and late-modal degradation. By analyzing the behavior of MLLMs at different stages, we propose a plateau-guided model merging method that selectively injects base language model parameters into MLLMs. Experimental results based on five MLLMs on nine benchmarks demonstrate the effectiveness of our method. Attention-based analysis further reveals that merging shifts attention from diffuse, scattered patterns to focused localization on task-relevant visual regions. Our repository is on https://github.com/wzj1718/PlaM. |
| title | PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.07645 |