Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918163665387520 |
|---|---|
| author | Hua, Yongxiang Cao, Haoyu Tao, Zhou Li, Bocheng Wu, Zihao Liu, Chaohu Xu, Linli |
| author_facet | Hua, Yongxiang Cao, Haoyu Tao, Zhou Li, Bocheng Wu, Zihao Liu, Chaohu Xu, Linli |
| contents | Sparse Mixture of Experts (sMoE) has become a pivotal approach for scaling large vision-language models, offering substantial capacity while maintaining computational efficiency through dynamic, sparse activation of experts. However, existing routing mechanisms, typically based on similarity scoring, struggle to effectively capture the underlying input structure. This limitation leads to a trade-off between expert specialization and balanced computation, hindering both scalability and performance. We propose Input Domain Aware MoE, a novel routing framework that leverages a probabilistic mixture model to better partition the input space. By modeling routing probabilities as a mixture of distributions, our method enables experts to develop clear specialization boundaries while achieving balanced utilization. Unlike conventional approaches, our routing mechanism is trained independently of task-specific objectives, allowing for stable optimization and decisive expert assignments. Empirical results on vision-language tasks demonstrate that our method consistently outperforms existing sMoE approaches, achieving higher task performance and improved expert utilization balance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_16448 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts Hua, Yongxiang Cao, Haoyu Tao, Zhou Li, Bocheng Wu, Zihao Liu, Chaohu Xu, Linli Machine Learning Artificial Intelligence Sparse Mixture of Experts (sMoE) has become a pivotal approach for scaling large vision-language models, offering substantial capacity while maintaining computational efficiency through dynamic, sparse activation of experts. However, existing routing mechanisms, typically based on similarity scoring, struggle to effectively capture the underlying input structure. This limitation leads to a trade-off between expert specialization and balanced computation, hindering both scalability and performance. We propose Input Domain Aware MoE, a novel routing framework that leverages a probabilistic mixture model to better partition the input space. By modeling routing probabilities as a mixture of distributions, our method enables experts to develop clear specialization boundaries while achieving balanced utilization. Unlike conventional approaches, our routing mechanism is trained independently of task-specific objectives, allowing for stable optimization and decisive expert assignments. Empirical results on vision-language tasks demonstrate that our method consistently outperforms existing sMoE approaches, achieving higher task performance and improved expert utilization balance. |
| title | Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.16448 |