Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911284372439040 |
|---|---|
| author | Pan, Zecheng Chen, Zhikang Li, Ding Zhang, Min Cui, Sen Jin, Hongshuo Tao, Luqi Yang, Yi Ye, Deheng Zhang, Yu Zhu, Tingting Ren, Tianling |
| author_facet | Pan, Zecheng Chen, Zhikang Li, Ding Zhang, Min Cui, Sen Jin, Hongshuo Tao, Luqi Yang, Yi Ye, Deheng Zhang, Yu Zhu, Tingting Ren, Tianling |
| contents | Merging models fine-tuned for different tasks into a single unified model has become an increasingly important direction for building versatile, efficient multi-task systems. Existing approaches predominantly rely on parameter interpolation in weight space, which we show introduces significant distribution shift in the feature space and undermines task-specific knowledge. In this paper, we propose OTMF (Optimal Transport-based Masked Fusion), a novel model merging framework rooted in optimal transport theory to address the distribution shift that arises from naive parameter interpolation. Instead of directly aggregating features or weights, OTMF aligns the semantic geometry of task-specific models by discovering common masks applied to task vectors through optimal transport plans. These masks selectively extract transferable and task-agnostic components while preserving the unique structural identities of each task. To ensure scalability in real-world settings, OTMF further supports a continual fusion paradigm that incrementally integrates each new task vector without revisiting previous ones, maintaining a bounded memory footprint and enabling efficient fusion across a growing number of tasks. We conduct comprehensive experiments on multiple vision and language benchmarks, and results show that OTMF achieves state-of-the-art performance in terms of both accuracy and efficiency. These findings highlight the practical and theoretical value of our approach to model merging. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_19561 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport Pan, Zecheng Chen, Zhikang Li, Ding Zhang, Min Cui, Sen Jin, Hongshuo Tao, Luqi Yang, Yi Ye, Deheng Zhang, Yu Zhu, Tingting Ren, Tianling Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Merging models fine-tuned for different tasks into a single unified model has become an increasingly important direction for building versatile, efficient multi-task systems. Existing approaches predominantly rely on parameter interpolation in weight space, which we show introduces significant distribution shift in the feature space and undermines task-specific knowledge. In this paper, we propose OTMF (Optimal Transport-based Masked Fusion), a novel model merging framework rooted in optimal transport theory to address the distribution shift that arises from naive parameter interpolation. Instead of directly aggregating features or weights, OTMF aligns the semantic geometry of task-specific models by discovering common masks applied to task vectors through optimal transport plans. These masks selectively extract transferable and task-agnostic components while preserving the unique structural identities of each task. To ensure scalability in real-world settings, OTMF further supports a continual fusion paradigm that incrementally integrates each new task vector without revisiting previous ones, maintaining a bounded memory footprint and enabling efficient fusion across a growing number of tasks. We conduct comprehensive experiments on multiple vision and language benchmarks, and results show that OTMF achieves state-of-the-art performance in terms of both accuracy and efficiency. These findings highlight the practical and theoretical value of our approach to model merging. |
| title | Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.19561 |