Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.18586 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912392398503936 |
|---|---|
| author | Min, Chengxi Wang, Wei Liu, Yahui Ye, Weixin Sangineto, Enver Wang, Qi Zhao, Yao |
| author_facet | Min, Chengxi Wang, Wei Liu, Yahui Ye, Weixin Sangineto, Enver Wang, Qi Zhao, Yao |
| contents | Mixture-of-Experts (MoE) models have emerged as a promising direction for scaling vision architectures efficiently. Among them, Soft MoE improves training stability by assigning each token to all experts via continuous dispatch weights. However, current designs overlook the semantic structure which is implicitly encoded in these weights, resulting in suboptimal expert routing. In this paper, we discover that dispatch weights in Soft MoE inherently exhibit segmentation-like patterns but are not explicitly aligned with semantic regions. Motivated by this observation, we propose a foreground-guided enhancement strategy. Specifically, we introduce a spatially aware auxiliary loss that encourages expert activation to align with semantic foreground regions. To further reinforce this supervision, we integrate a lightweight LayerScale mechanism that improves information flow and stabilizes optimization in skip connections. Our method necessitates only minor architectural adjustments and can be seamlessly integrated into prevailing Soft MoE frameworks. Comprehensive experiments on ImageNet-1K and multiple smaller-scale classification benchmarks not only showcase consistent performance enhancements but also reveal more interpretable expert routing mechanisms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18586 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing Min, Chengxi Wang, Wei Liu, Yahui Ye, Weixin Sangineto, Enver Wang, Qi Zhao, Yao Computer Vision and Pattern Recognition Mixture-of-Experts (MoE) models have emerged as a promising direction for scaling vision architectures efficiently. Among them, Soft MoE improves training stability by assigning each token to all experts via continuous dispatch weights. However, current designs overlook the semantic structure which is implicitly encoded in these weights, resulting in suboptimal expert routing. In this paper, we discover that dispatch weights in Soft MoE inherently exhibit segmentation-like patterns but are not explicitly aligned with semantic regions. Motivated by this observation, we propose a foreground-guided enhancement strategy. Specifically, we introduce a spatially aware auxiliary loss that encourages expert activation to align with semantic foreground regions. To further reinforce this supervision, we integrate a lightweight LayerScale mechanism that improves information flow and stabilizes optimization in skip connections. Our method necessitates only minor architectural adjustments and can be seamlessly integrated into prevailing Soft MoE frameworks. Comprehensive experiments on ImageNet-1K and multiple smaller-scale classification benchmarks not only showcase consistent performance enhancements but also reveal more interpretable expert routing mechanisms. |
| title | Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.18586 |