Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoE
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911308415238144 |
|---|---|
| author | Zeng, Anxiang Zhang, Haibo Zhang, Hailing Mo, Kaixiang Yao, Liang Hu, Ling Zhang, Long Liu, Shuman Xie, Shuyi Li, Yanshi Chen, Yizhang Sheng, Yuepeng Huang, Yuwei Xu, Zhaochen Zhou, Zhiqiang Liew, Ziqin |
| author_facet | Zeng, Anxiang Zhang, Haibo Zhang, Hailing Mo, Kaixiang Yao, Liang Hu, Ling Zhang, Long Liu, Shuman Xie, Shuyi Li, Yanshi Chen, Yizhang Sheng, Yuepeng Huang, Yuwei Xu, Zhaochen Zhou, Zhiqiang Liew, Ziqin |
| contents | We present CompassMax-V3-Thinking, a hundred-billion-scale MoE reasoning model trained with a new RL framework built on one principle: each prompt must matter. Scaling RL to this size exposes critical inefficiencies-zero-variance prompts that waste rollouts, unstable importance sampling over long horizons, advantage inversion from standard reward models, and systemic bottlenecks in rollout processing. To overcome these challenges, we introduce several unified innovations: (1) Multi-Stage Zero-Variance Elimination, which filters out non-informative prompts and stabilizes group-based policy optimization (e.g. GRPO) by removing wasted rollouts; (2) ESPO, an entropy-adaptive optimization method that balances token-level and sequence-level importance sampling to maintain stable learning dynamics; (3) a Router Replay strategy that aligns training-time MoE router decisions with inference-time behavior to mitigate train-infer discrepancies, coupled with a reward model adjustment to prevent advantage inversion; (4) a high-throughput RL system with FP8-precision rollouts, overlapped reward computation, and length-aware scheduling to eliminate performance bottlenecks. Together, these contributions form a cohesive pipeline that makes RL on hundred-billion-scale MoE models stable and efficient. The resulting model delivers strong performance across both internal and public evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_07710 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoE Zeng, Anxiang Zhang, Haibo Zhang, Hailing Mo, Kaixiang Yao, Liang Hu, Ling Zhang, Long Liu, Shuman Xie, Shuyi Li, Yanshi Chen, Yizhang Sheng, Yuepeng Huang, Yuwei Xu, Zhaochen Zhou, Zhiqiang Liew, Ziqin Artificial Intelligence Machine Learning We present CompassMax-V3-Thinking, a hundred-billion-scale MoE reasoning model trained with a new RL framework built on one principle: each prompt must matter. Scaling RL to this size exposes critical inefficiencies-zero-variance prompts that waste rollouts, unstable importance sampling over long horizons, advantage inversion from standard reward models, and systemic bottlenecks in rollout processing. To overcome these challenges, we introduce several unified innovations: (1) Multi-Stage Zero-Variance Elimination, which filters out non-informative prompts and stabilizes group-based policy optimization (e.g. GRPO) by removing wasted rollouts; (2) ESPO, an entropy-adaptive optimization method that balances token-level and sequence-level importance sampling to maintain stable learning dynamics; (3) a Router Replay strategy that aligns training-time MoE router decisions with inference-time behavior to mitigate train-infer discrepancies, coupled with a reward model adjustment to prevent advantage inversion; (4) a high-throughput RL system with FP8-precision rollouts, overlapped reward computation, and length-aware scheduling to eliminate performance bottlenecks. Together, these contributions form a cohesive pipeline that makes RL on hundred-billion-scale MoE models stable and efficient. The resulting model delivers strong performance across both internal and public evaluations. |
| title | Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoE |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2512.07710 |