Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917444511072256 |
|---|---|
| author | Hu, Wentao Zhai, Yanbo Hu, Xiaohui Zhao, Mingkuan yu, Shanhong Liu, Xue Yu, Kaidong Song, Shuangyong Li, Xuelong |
| author_facet | Hu, Wentao Zhai, Yanbo Hu, Xiaohui Zhao, Mingkuan yu, Shanhong Liu, Xue Yu, Kaidong Song, Shuangyong Li, Xuelong |
| contents | Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We identify that this fragility stems from static Top-$k$ routing: routers tend to favor high-frequency patterns over rare factual associations. Consequently, ``specialist experts'' possessing critical long-tail knowledge are often assigned low gating scores and remain ``dormant'' -- under-prioritized for specific tokens despite their proven causal importance on other inputs. To address this, we propose Counterfactual Routing (CoR), a training-free inference framework designed to awaken these dormant experts. CoR integrates layer-wise perturbation analysis with the Counterfactual Expert Impact (CEI) metric to dynamically shift computational resources from syntax-dominant to knowledge-intensive layers while maintaining a constant total activation count, effectively retrieving causally decisive experts via virtual ablation. Extensive experiments on TruthfulQA, FACTOR, and TriviaQA demonstrate that CoR improves factual accuracy by 3.1\% on average without increasing the inference budget, establishing a superior Pareto frontier compared to static scaling strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_14246 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations Hu, Wentao Zhai, Yanbo Hu, Xiaohui Zhao, Mingkuan yu, Shanhong Liu, Xue Yu, Kaidong Song, Shuangyong Li, Xuelong Machine Learning Artificial Intelligence Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We identify that this fragility stems from static Top-$k$ routing: routers tend to favor high-frequency patterns over rare factual associations. Consequently, ``specialist experts'' possessing critical long-tail knowledge are often assigned low gating scores and remain ``dormant'' -- under-prioritized for specific tokens despite their proven causal importance on other inputs. To address this, we propose Counterfactual Routing (CoR), a training-free inference framework designed to awaken these dormant experts. CoR integrates layer-wise perturbation analysis with the Counterfactual Expert Impact (CEI) metric to dynamically shift computational resources from syntax-dominant to knowledge-intensive layers while maintaining a constant total activation count, effectively retrieving causally decisive experts via virtual ablation. Extensive experiments on TruthfulQA, FACTOR, and TriviaQA demonstrate that CoR improves factual accuracy by 3.1\% on average without increasing the inference budget, establishing a superior Pareto frontier compared to static scaling strategies. |
| title | Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.14246 |