Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918282495262720 |
|---|---|
| author | Zhang, Di Wu, Xun Huang, Shaohan Jiang, Lingjie Hao, Yaru Dong, Li Chi, Zewen Sui, Zhifang Wei, Furu |
| author_facet | Zhang, Di Wu, Xun Huang, Shaohan Jiang, Lingjie Hao, Yaru Dong, Li Chi, Zewen Sui, Zhifang Wei, Furu |
| contents | Recent advances in reinforcement learning (RL) have substantially improved the training of large-scale language models, leading to significant gains in generation quality and reasoning ability. However, most existing research focuses on dense models, while RL training for Mixture-of-Experts (MoE) architectures remains underexplored. To address the instability commonly observed in MoE training, we propose a novel router-aware approach to optimize importance sampling (IS) weights in off-policy RL. Specifically, we design a rescaling strategy guided by router logits, which effectively reduces gradient variance and mitigates training divergence. Experimental results demonstrate that our method significantly improves both the convergence stability and the final performance of MoE models, highlighting the potential of RL algorithmic innovations tailored to MoE architectures and providing a promising direction for efficient training of large-scale expert models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_23027 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts Zhang, Di Wu, Xun Huang, Shaohan Jiang, Lingjie Hao, Yaru Dong, Li Chi, Zewen Sui, Zhifang Wei, Furu Machine Learning Computation and Language Recent advances in reinforcement learning (RL) have substantially improved the training of large-scale language models, leading to significant gains in generation quality and reasoning ability. However, most existing research focuses on dense models, while RL training for Mixture-of-Experts (MoE) architectures remains underexplored. To address the instability commonly observed in MoE training, we propose a novel router-aware approach to optimize importance sampling (IS) weights in off-policy RL. Specifically, we design a rescaling strategy guided by router logits, which effectively reduces gradient variance and mitigates training divergence. Experimental results demonstrate that our method significantly improves both the convergence stability and the final performance of MoE models, highlighting the potential of RL algorithmic innovations tailored to MoE architectures and providing a promising direction for efficient training of large-scale expert models. |
| title | Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2510.23027 |