Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.17110 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909620820246528 |
|---|---|
| author | Li, Junlin DU, Guodong Li, Jing Goh, Sim Kuan Wang, Wenya Wang, Yequan Liu, Fangming Tang, Ho-Kin Alharbi, Saleh He, Daojing Zhang, Min |
| author_facet | Li, Junlin DU, Guodong Li, Jing Goh, Sim Kuan Wang, Wenya Wang, Yequan Liu, Fangming Tang, Ho-Kin Alharbi, Saleh He, Daojing Zhang, Min |
| contents | Fine-tuning Large Language Models (LLMs) with multimodal encoders on modality-specific data expands the modalities that LLMs can handle, leading to the formation of Multimodal LLMs (MLLMs). However, this paradigm heavily relies on resource-intensive and inflexible fine-tuning from scratch with new multimodal data. In this paper, we propose MMER (Multi-modality Expansion and Retention), a training-free approach that integrates existing MLLMs for effective multimodal expansion while retaining their original performance. Specifically, MMER reuses MLLMs' multimodal encoders while merging their LLM parameters. By comparing original and merged LLM parameters, MMER generates binary masks to approximately separate LLM parameters for each modality. These decoupled parameters can independently process modality-specific inputs, reducing parameter conflicts and preserving original MLLMs' fidelity. MMER can also mitigate catastrophic forgetting by applying a similar process to MLLMs fine-tuned on new tasks. Extensive experiments show significant improvements over baselines, proving that MMER effectively expands LLMs' multimodal capabilities while retaining 99% of the original performance, and also markedly mitigates catastrophic forgetting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17110 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling Li, Junlin DU, Guodong Li, Jing Goh, Sim Kuan Wang, Wenya Wang, Yequan Liu, Fangming Tang, Ho-Kin Alharbi, Saleh He, Daojing Zhang, Min Computation and Language Fine-tuning Large Language Models (LLMs) with multimodal encoders on modality-specific data expands the modalities that LLMs can handle, leading to the formation of Multimodal LLMs (MLLMs). However, this paradigm heavily relies on resource-intensive and inflexible fine-tuning from scratch with new multimodal data. In this paper, we propose MMER (Multi-modality Expansion and Retention), a training-free approach that integrates existing MLLMs for effective multimodal expansion while retaining their original performance. Specifically, MMER reuses MLLMs' multimodal encoders while merging their LLM parameters. By comparing original and merged LLM parameters, MMER generates binary masks to approximately separate LLM parameters for each modality. These decoupled parameters can independently process modality-specific inputs, reducing parameter conflicts and preserving original MLLMs' fidelity. MMER can also mitigate catastrophic forgetting by applying a similar process to MLLMs fine-tuned on new tasks. Extensive experiments show significant improvements over baselines, proving that MMER effectively expands LLMs' multimodal capabilities while retaining 99% of the original performance, and also markedly mitigates catastrophic forgetting. |
| title | Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.17110 |