Post-Trained MoE Can Skip Half Experts via Self-Distillation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909054497980416 |
|---|---|
| author | Lv, Xingtai Sheng, Li Zhang, Kaiyan You, Yichen Gao, Siyan Luo, Xueheng Zuo, Yuxin Fan, Yuchen Yang, Junlin Cui, Ganqu Wang, Bingning Yang, Fan Sun, Youbang Ding, Ning Zhou, Bowen |
| author_facet | Lv, Xingtai Sheng, Li Zhang, Kaiyan You, Yichen Gao, Siyan Luo, Xueheng Zuo, Yuxin Fan, Yuchen Yang, Junlin Cui, Ganqu Wang, Bingning Yang, Fan Sun, Youbang Ding, Ning Zhou, Bowen |
| contents | Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation, and its dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner. Existing dynamic MoE methods usually rely on pre-training from scratch or task-specific adaptation, leaving the practical conversion of fully trained MoE underexplored. Enabling such adaptation would directly alleviate the inference costs by allowing easy tokens to bypass unnecessary expert during serving. This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones. To stabilize this architectural conversion, ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the augmented model through two-stage self-distillation, utilizing the original MoE as a frozen teacher and applying a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss. It outperforms the strongest dynamic MoE baseline by 6.1 and 4.0 points on the two models, and delivers ~1.20$\times$ end-to-end inference speedup. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_18643 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Post-Trained MoE Can Skip Half Experts via Self-Distillation Lv, Xingtai Sheng, Li Zhang, Kaiyan You, Yichen Gao, Siyan Luo, Xueheng Zuo, Yuxin Fan, Yuchen Yang, Junlin Cui, Ganqu Wang, Bingning Yang, Fan Sun, Youbang Ding, Ning Zhou, Bowen Machine Learning Artificial Intelligence Computation and Language Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation, and its dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner. Existing dynamic MoE methods usually rely on pre-training from scratch or task-specific adaptation, leaving the practical conversion of fully trained MoE underexplored. Enabling such adaptation would directly alleviate the inference costs by allowing easy tokens to bypass unnecessary expert during serving. This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones. To stabilize this architectural conversion, ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the augmented model through two-stage self-distillation, utilizing the original MoE as a frozen teacher and applying a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss. It outperforms the strongest dynamic MoE baseline by 6.1 and 4.0 points on the two models, and delivers ~1.20$\times$ end-to-end inference speedup. |
| title | Post-Trained MoE Can Skip Half Experts via Self-Distillation |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2605.18643 |