Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.14731 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909651725975552 |
|---|---|
| author | Ling Team Hu, Bin Chen, Cai Zhao, Deng Liu, Ding Jin, Dingnan Zhu, Feng Dai, Hao Luan, Hongzhi Guo, Jia Liu, Jiaming Wu, Jiewei Mei, Jun Zhou, Jun Zhao, Junbo Xiong, Junwu Zhang, Kaihong Xu, Kuan Liang, Lei Jiang, Liang Fu, Liangcheng Zheng, Longfei Gao, Qiang Cui, Qing Wan, Quan Zheng, Shaomian Li, Shuaicheng Yang, Tongkai Ren, Wang Yan, Xiaodong Wan, Xiaopei Feng, Xiaoyun Zhao, Xin Yang, Xinxing Kong, Xinyu Yang, Xuemin Li, Yang Wu, Yingting Liu, Yongkang Xu, Zhankai Zhang, Zhenduo Zhou, Zhenglei Huang, Zhenyu Zhang, Zhiqiang Wang, Zihao Wen, Zujie |
| author_facet | Ling Team Hu, Bin Chen, Cai Zhao, Deng Liu, Ding Jin, Dingnan Zhu, Feng Dai, Hao Luan, Hongzhi Guo, Jia Liu, Jiaming Wu, Jiewei Mei, Jun Zhou, Jun Zhao, Junbo Xiong, Junwu Zhang, Kaihong Xu, Kuan Liang, Lei Jiang, Liang Fu, Liangcheng Zheng, Longfei Gao, Qiang Cui, Qing Wan, Quan Zheng, Shaomian Li, Shuaicheng Yang, Tongkai Ren, Wang Yan, Xiaodong Wan, Xiaopei Feng, Xiaoyun Zhao, Xin Yang, Xinxing Kong, Xinyu Yang, Xuemin Li, Yang Wu, Yingting Liu, Yongkang Xu, Zhankai Zhang, Zhenduo Zhou, Zhenglei Huang, Zhenyu Zhang, Zhiqiang Wang, Zihao Wen, Zujie |
| contents | We present Ring-lite, a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL) to achieve efficient and robust reasoning capabilities. Built upon the publicly available Ling-lite model, a 16.8 billion parameter model with 2.75 billion activated parameters, our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks (e.g., AIME, LiveCodeBench, GPQA-Diamond) while activating only one-third of the parameters required by comparable models. To accomplish this, we introduce a joint training pipeline integrating distillation with RL, revealing undocumented challenges in MoE RL training. First, we identify optimization instability during RL training, and we propose Constrained Contextual Computation Policy Optimization(C3PO), a novel approach that enhances training stability and improves computational throughput via algorithm-system co-design methodology. Second, we empirically demonstrate that selecting distillation checkpoints based on entropy loss for RL training, rather than validation metrics, yields superior performance-efficiency trade-offs in subsequent RL training. Finally, we develop a two-stage training paradigm to harmonize multi-domain data integration, addressing domain conflicts that arise in training with mixed dataset. We will release the model, dataset, and code. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14731 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs Ling Team Hu, Bin Chen, Cai Zhao, Deng Liu, Ding Jin, Dingnan Zhu, Feng Dai, Hao Luan, Hongzhi Guo, Jia Liu, Jiaming Wu, Jiewei Mei, Jun Zhou, Jun Zhao, Junbo Xiong, Junwu Zhang, Kaihong Xu, Kuan Liang, Lei Jiang, Liang Fu, Liangcheng Zheng, Longfei Gao, Qiang Cui, Qing Wan, Quan Zheng, Shaomian Li, Shuaicheng Yang, Tongkai Ren, Wang Yan, Xiaodong Wan, Xiaopei Feng, Xiaoyun Zhao, Xin Yang, Xinxing Kong, Xinyu Yang, Xuemin Li, Yang Wu, Yingting Liu, Yongkang Xu, Zhankai Zhang, Zhenduo Zhou, Zhenglei Huang, Zhenyu Zhang, Zhiqiang Wang, Zihao Wen, Zujie Computation and Language Artificial Intelligence We present Ring-lite, a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL) to achieve efficient and robust reasoning capabilities. Built upon the publicly available Ling-lite model, a 16.8 billion parameter model with 2.75 billion activated parameters, our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks (e.g., AIME, LiveCodeBench, GPQA-Diamond) while activating only one-third of the parameters required by comparable models. To accomplish this, we introduce a joint training pipeline integrating distillation with RL, revealing undocumented challenges in MoE RL training. First, we identify optimization instability during RL training, and we propose Constrained Contextual Computation Policy Optimization(C3PO), a novel approach that enhances training stability and improves computational throughput via algorithm-system co-design methodology. Second, we empirically demonstrate that selecting distillation checkpoints based on entropy loss for RL training, rather than validation metrics, yields superior performance-efficiency trade-offs in subsequent RL training. Finally, we develop a two-stage training paradigm to harmonize multi-domain data integration, addressing domain conflicts that arise in training with mixed dataset. We will release the model, dataset, and code. |
| title | Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.14731 |