MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917152869580800 |
|---|---|
| author | Jia, Weitao Lu, Jinghui Yu, Haiyang Wang, Siqi Tang, Guozhi Wang, An-Lan Yin, Weijie Yang, Dingkang Nie, Yuxiang Shan, Bin Feng, Hao Li, Irene Yang, Kun Wang, Han Tang, Jingqun Fu, Teng Jin, Changhong Feng, Chao Lv, Xiaohui Huang, Can |
| author_facet | Jia, Weitao Lu, Jinghui Yu, Haiyang Wang, Siqi Tang, Guozhi Wang, An-Lan Yin, Weijie Yang, Dingkang Nie, Yuxiang Shan, Bin Feng, Hao Li, Irene Yang, Kun Wang, Han Tang, Jingqun Fu, Teng Jin, Changhong Feng, Chao Lv, Xiaohui Huang, Can |
| contents | Recent advances demonstrate that reinforcement learning with verifiable rewards (RLVR) significantly enhances the reasoning capabilities of large language models (LLMs). However, standard RLVR faces challenges with reward sparsity, where zero rewards from consistently incorrect candidate answers provide no learning signal, particularly in challenging tasks. To address this, we propose Multi-Expert Mutual Learning GRPO (MEML-GRPO), an innovative framework that utilizes diverse expert prompts as system prompts to generate a broader range of responses, substantially increasing the likelihood of identifying correct solutions. Additionally, we introduce an inter-expert mutual learning mechanism that facilitates knowledge sharing and transfer among experts, further boosting the model's performance through RLVR. Extensive experiments across multiple reasoning benchmarks show that MEML-GRPO delivers significant improvements, achieving an average performance gain of 4.89% with Qwen and 11.33% with Llama, effectively overcoming the core limitations of traditional RLVR methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09670 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement Jia, Weitao Lu, Jinghui Yu, Haiyang Wang, Siqi Tang, Guozhi Wang, An-Lan Yin, Weijie Yang, Dingkang Nie, Yuxiang Shan, Bin Feng, Hao Li, Irene Yang, Kun Wang, Han Tang, Jingqun Fu, Teng Jin, Changhong Feng, Chao Lv, Xiaohui Huang, Can Artificial Intelligence Recent advances demonstrate that reinforcement learning with verifiable rewards (RLVR) significantly enhances the reasoning capabilities of large language models (LLMs). However, standard RLVR faces challenges with reward sparsity, where zero rewards from consistently incorrect candidate answers provide no learning signal, particularly in challenging tasks. To address this, we propose Multi-Expert Mutual Learning GRPO (MEML-GRPO), an innovative framework that utilizes diverse expert prompts as system prompts to generate a broader range of responses, substantially increasing the likelihood of identifying correct solutions. Additionally, we introduce an inter-expert mutual learning mechanism that facilitates knowledge sharing and transfer among experts, further boosting the model's performance through RLVR. Extensive experiments across multiple reasoning benchmarks show that MEML-GRPO delivers significant improvements, achieving an average performance gain of 4.89% with Qwen and 11.33% with Llama, effectively overcoming the core limitations of traditional RLVR methods. |
| title | MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2508.09670 |