Expert Merging in Sparse Mixture of Experts with Nash Bargaining
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914615078682624 |
|---|---|
| author | Nguyen, Dung V. Nguyen, Anh T. Nguyen, Minh H. Nguyen, Luc Q. Jiang, Shiqi Fetaya, Ethan Tran, Linh Duy Chechik, Gal Nguyen, Tan M. |
| author_facet | Nguyen, Dung V. Nguyen, Anh T. Nguyen, Minh H. Nguyen, Luc Q. Jiang, Shiqi Fetaya, Ethan Tran, Linh Duy Chechik, Gal Nguyen, Tan M. |
| contents | Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings. The code is publicly available at: https://github.com/anh147/NAMEx. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_16138 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Expert Merging in Sparse Mixture of Experts with Nash Bargaining Nguyen, Dung V. Nguyen, Anh T. Nguyen, Minh H. Nguyen, Luc Q. Jiang, Shiqi Fetaya, Ethan Tran, Linh Duy Chechik, Gal Nguyen, Tan M. Machine Learning Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings. The code is publicly available at: https://github.com/anh147/NAMEx. |
| title | Expert Merging in Sparse Mixture of Experts with Nash Bargaining |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.16138 |