SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918469414420480 |
|---|---|
| author | Bo, Zi-Hao Li, Yaqian Hou, Anzhou Takezoe, Rinyoichi Zhao, Ertao Pan, Tianxiang Yan, Jiale Guang, Mo Long, Kaiwen |
| author_facet | Bo, Zi-Hao Li, Yaqian Hou, Anzhou Takezoe, Rinyoichi Zhao, Ertao Pan, Tianxiang Yan, Jiale Guang, Mo Long, Kaiwen |
| contents | Mixture-of-Experts (MoE) has become a prevalent backbone for large vision-language models (VLMs), yet how modality-specific signals should guide expert routing remains under-explored. Existing routing strategies are either hand-crafted or modality-agnostic, relying on idealized priors that ignore the layer-dependent modality fusion patterns in MoE-VLMs and provide little guidance for expert specialization. We propose Soft Modality-guided Expert Specialization (SMoES), which consists of dynamic soft modality scores that capture layer-dependent fusion patterns, an expert binning mechanism aligned with expert-parallel deployment, and an inter-bin mutual information regularization that encourages coherent modality specialization. Our method leverages attention-based or Gaussian-statistics modality scores to optimize mutual information regularization. Experiments across four MoE-based VLMs and 16 benchmarks demonstrate improvement on both effectiveness and efficiency: 0.9% and 4.2% average gain on multimodal and language-only tasks, 56.1% reduction in EP communication overhead, and 12.3% throughput improvement under realistic deployment. These results validate that aligning routing with modality-aware expert specialization unlocks MoE-VLM capacity and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_23996 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs Bo, Zi-Hao Li, Yaqian Hou, Anzhou Takezoe, Rinyoichi Zhao, Ertao Pan, Tianxiang Yan, Jiale Guang, Mo Long, Kaiwen Computer Vision and Pattern Recognition Mixture-of-Experts (MoE) has become a prevalent backbone for large vision-language models (VLMs), yet how modality-specific signals should guide expert routing remains under-explored. Existing routing strategies are either hand-crafted or modality-agnostic, relying on idealized priors that ignore the layer-dependent modality fusion patterns in MoE-VLMs and provide little guidance for expert specialization. We propose Soft Modality-guided Expert Specialization (SMoES), which consists of dynamic soft modality scores that capture layer-dependent fusion patterns, an expert binning mechanism aligned with expert-parallel deployment, and an inter-bin mutual information regularization that encourages coherent modality specialization. Our method leverages attention-based or Gaussian-statistics modality scores to optimize mutual information regularization. Experiments across four MoE-based VLMs and 16 benchmarks demonstrate improvement on both effectiveness and efficiency: 0.9% and 4.2% average gain on multimodal and language-only tasks, 56.1% reduction in EP communication overhead, and 12.3% throughput improvement under realistic deployment. These results validate that aligning routing with modality-aware expert specialization unlocks MoE-VLM capacity and efficiency. |
| title | SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.23996 |