MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913693771497472 |
|---|---|
| author | Zhou, Yuhang Karamanolakis, Giannis Soto, Victor Rumshisky, Anna Kulkarni, Mayank Huang, Furong Ai, Wei Lu, Jianhua |
| author_facet | Zhou, Yuhang Karamanolakis, Giannis Soto, Victor Rumshisky, Anna Kulkarni, Mayank Huang, Furong Ai, Wei Lu, Jianhua |
| contents | The recent success of specialized Large Language Models (LLMs) in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert LLMs into a unified Mixture-of-Experts (MoE) model, with the goal of enhancing performance in each domain while retaining effectiveness on general tasks. However, the effective merging of expert models remains an open challenge, especially for models with highly divergent weight parameters or different architectures. State-of-the-art MoE merging methods only work with homogeneous model architectures and rely on simple unweighted averaging to merge expert layers, which does not address parameter interference and requires extensive fine-tuning of the merged MoE to restore performance. To address these limitations, this paper introduces new MoE merging techniques, including strategies to mitigate parameter interference, routing heuristics to reduce the need for MoE fine-tuning, and a novel method for merging experts with different architectures. Extensive experiments across multiple domains demonstrate the effectiveness of our proposed methods, reducing fine-tuning costs, improving performance over state-of-the-art methods, and expanding the applicability of MoE merging. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00997 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs Zhou, Yuhang Karamanolakis, Giannis Soto, Victor Rumshisky, Anna Kulkarni, Mayank Huang, Furong Ai, Wei Lu, Jianhua Computation and Language Artificial Intelligence The recent success of specialized Large Language Models (LLMs) in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert LLMs into a unified Mixture-of-Experts (MoE) model, with the goal of enhancing performance in each domain while retaining effectiveness on general tasks. However, the effective merging of expert models remains an open challenge, especially for models with highly divergent weight parameters or different architectures. State-of-the-art MoE merging methods only work with homogeneous model architectures and rely on simple unweighted averaging to merge expert layers, which does not address parameter interference and requires extensive fine-tuning of the merged MoE to restore performance. To address these limitations, this paper introduces new MoE merging techniques, including strategies to mitigate parameter interference, routing heuristics to reduce the need for MoE fine-tuning, and a novel method for merging experts with different architectures. Extensive experiments across multiple domains demonstrate the effectiveness of our proposed methods, reducing fine-tuning costs, improving performance over state-of-the-art methods, and expanding the applicability of MoE merging. |
| title | MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.00997 |