Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914101028978688 |
|---|---|
| author | Nguyen-Nhat, Minh-Khoi Teo, Rachel S. Y. Abdullaev, Laziz Mok, Maurice Tran, Viet-Hoang Nguyen, Tan Minh |
| author_facet | Nguyen-Nhat, Minh-Khoi Teo, Rachel S. Y. Abdullaev, Laziz Mok, Maurice Tran, Viet-Hoang Nguyen, Tan Minh |
| contents | Sparse Mixture of Experts (SMoE) has emerged as a promising solution to achieving unparalleled scalability in deep learning by decoupling model parameter count from computational cost. By activating only a small subset of parameters per sample, SMoE enables significant growth in model capacity while maintaining efficiency. However, SMoE struggles to adapt to distributional shifts, leading to reduced robustness under data contamination. In this work, we introduce SymphonySMoE, a novel family of SMoE that introduces a social graph to model interactions among experts. This graph-based structure enhances the token routing process, addressing the robustness challenges that are inherent in conventional SMoE designs. SymphonySMoE is lightweight, modular, and integrates seamlessly with existing SMoE-based models such as the XMoE and the Generalist Language Model. We provide both theoretical analysis and empirical evidence demonstrating SymphonySMoE's advantages over baseline SMoE. Extensive experiments on language modeling and visual instruction tuning validate our method's effectiveness. We further highlight the scalability of SymphonySMoE to models with 4.2 and 7.4 billion parameters, showcasing its applicability in fine-tuning tasks for large-scale systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_16411 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures Nguyen-Nhat, Minh-Khoi Teo, Rachel S. Y. Abdullaev, Laziz Mok, Maurice Tran, Viet-Hoang Nguyen, Tan Minh Machine Learning Artificial Intelligence Sparse Mixture of Experts (SMoE) has emerged as a promising solution to achieving unparalleled scalability in deep learning by decoupling model parameter count from computational cost. By activating only a small subset of parameters per sample, SMoE enables significant growth in model capacity while maintaining efficiency. However, SMoE struggles to adapt to distributional shifts, leading to reduced robustness under data contamination. In this work, we introduce SymphonySMoE, a novel family of SMoE that introduces a social graph to model interactions among experts. This graph-based structure enhances the token routing process, addressing the robustness challenges that are inherent in conventional SMoE designs. SymphonySMoE is lightweight, modular, and integrates seamlessly with existing SMoE-based models such as the XMoE and the Generalist Language Model. We provide both theoretical analysis and empirical evidence demonstrating SymphonySMoE's advantages over baseline SMoE. Extensive experiments on language modeling and visual instruction tuning validate our method's effectiveness. We further highlight the scalability of SymphonySMoE to models with 4.2 and 7.4 billion parameters, showcasing its applicability in fine-tuning tasks for large-scale systems. |
| title | Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.16411 |