Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913004806733824 |
|---|---|
| author | Zheng, Kening Huang, Wei-Chieh Huo, Jiahao Li, Zhonghao Zou, Henry Peng Yan, Yibo Zou, Xin Li, Jungang Li, Junzhuo Zhang, Hanrong Hu, Xuming Yu, Philip S. |
| author_facet | Zheng, Kening Huang, Wei-Chieh Huo, Jiahao Li, Zhonghao Zou, Henry Peng Yan, Yibo Zou, Xin Li, Jungang Li, Junzhuo Zhang, Hanrong Hu, Xuming Yu, Philip S. |
| contents | Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term Language Routing Isolation, in which high- and low-resource languages tend to activate largely disjoint expert sets. Through layer-stratified analysis, we further show that routing patterns exhibit a layer-wise convergence-divergence pattern across model depth. Building on these findings, we propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expert subnetworks. RISE applies a tripartite selection strategy, using specificity scores to identify language-specific experts in shallow and deep layers and overlap scores to select universal experts in middle layers. By training only the selected subnetwork while freezing all other parameters, RISE substantially improves low-resource language performance while preserving capabilities in other languages. Experiments on 10 languages demonstrate that RISE achieves target-language F1 gains of up to 10.85% with minimal cross-lingual degradation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_03592 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation Zheng, Kening Huang, Wei-Chieh Huo, Jiahao Li, Zhonghao Zou, Henry Peng Yan, Yibo Zou, Xin Li, Jungang Li, Junzhuo Zhang, Hanrong Hu, Xuming Yu, Philip S. Computation and Language Artificial Intelligence Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term Language Routing Isolation, in which high- and low-resource languages tend to activate largely disjoint expert sets. Through layer-stratified analysis, we further show that routing patterns exhibit a layer-wise convergence-divergence pattern across model depth. Building on these findings, we propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expert subnetworks. RISE applies a tripartite selection strategy, using specificity scores to identify language-specific experts in shallow and deep layers and overlap scores to select universal experts in middle layers. By training only the selected subnetwork while freezing all other parameters, RISE substantially improves low-resource language performance while preserving capabilities in other languages. Experiments on 10 languages demonstrate that RISE achieves target-language F1 gains of up to 10.85% with minimal cross-lingual degradation. |
| title | Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.03592 |