Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.12566 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912961024491520 |
|---|---|
| author | Wang, Haoqing Long, Xiang Li, Ziheng Xu, Yilong Li, Tingguang Tang, Yehui |
| author_facet | Wang, Haoqing Long, Xiang Li, Ziheng Xu, Yilong Li, Tingguang Tang, Yehui |
| contents | Reinforcement Learning with Verifiable Rewards (RLVR) plays a key role in stimulating the explicit reasoning capability of Large Language Models (LLMs). We can achieve expert-level performance in some specific domains via RLVR, such as coding or math. When a general multi-domain expert-level model is required, we need to carefully consider the collaboration of RLVR across different domains. The current state-of-the-art models mainly employ two different training paradigms for multi-domain RLVR: mixed multi-task RLVR and separate RLVR followed by model merging. However, most of the works did not provide a detailed comparison and analysis about these paradigms. To this end, we choose multiple commonly used high-level tasks (e.g., math, coding, science, instruction following, and agent) as our target domains and design extensive qualitative and quantitative experiments using open-source datasets. We find the RLVR across domains exhibits few mutual interferences, and reasoning-intensive domains demonstrate mutually synergistic effects. Furthermore, we analyze the internal mechanisms of mutual gains from the perspectives of weight space geometry, information constraints, model prediction behavior and self-verification. This project is named as M2RL that means Mixed multi-task training or separate training followed by model Merging for Reinforcement Learning, and the homepage is at https://github.com/Mosi-AI/M2RL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_12566 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models Wang, Haoqing Long, Xiang Li, Ziheng Xu, Yilong Li, Tingguang Tang, Yehui Artificial Intelligence Reinforcement Learning with Verifiable Rewards (RLVR) plays a key role in stimulating the explicit reasoning capability of Large Language Models (LLMs). We can achieve expert-level performance in some specific domains via RLVR, such as coding or math. When a general multi-domain expert-level model is required, we need to carefully consider the collaboration of RLVR across different domains. The current state-of-the-art models mainly employ two different training paradigms for multi-domain RLVR: mixed multi-task RLVR and separate RLVR followed by model merging. However, most of the works did not provide a detailed comparison and analysis about these paradigms. To this end, we choose multiple commonly used high-level tasks (e.g., math, coding, science, instruction following, and agent) as our target domains and design extensive qualitative and quantitative experiments using open-source datasets. We find the RLVR across domains exhibits few mutual interferences, and reasoning-intensive domains demonstrate mutually synergistic effects. Furthermore, we analyze the internal mechanisms of mutual gains from the perspectives of weight space geometry, information constraints, model prediction behavior and self-verification. This project is named as M2RL that means Mixed multi-task training or separate training followed by model Merging for Reinforcement Learning, and the homepage is at https://github.com/Mosi-AI/M2RL. |
| title | To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2602.12566 |