MARTI-MARS$^2$: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908820097204224 |
|---|---|
| author | Wang, Shijie Li, Pengfei Fu, Yikun Liu, Kaifeng Li, Fangyuan Liu, Yang Sun, Xiaowei Li, Zonglin Zhao, Siyao Zhao, Jian Tian, Kai Li, Dong Gao, Junqi Zhang, Yutong Chen, Yiqun Li, Yuqiang Li, Zoe Zhang, Weinan Ye, Peng Hu, Shuyue Bai, Lei Zhou, Bowen Zhang, Kaiyan Qi, Biqing |
| author_facet | Wang, Shijie Li, Pengfei Fu, Yikun Liu, Kaifeng Li, Fangyuan Liu, Yang Sun, Xiaowei Li, Zonglin Zhao, Siyao Zhao, Jian Tian, Kai Li, Dong Gao, Junqi Zhang, Yutong Chen, Yiqun Li, Yuqiang Li, Zoe Zhang, Weinan Ye, Peng Hu, Shuyue Bai, Lei Zhou, Bowen Zhang, Kaiyan Qi, Biqing |
| contents | While the complex reasoning capability of Large Language Models (LLMs) has attracted significant attention, single-agent systems often encounter inherent performance ceilings in complex tasks such as code generation. Multi-agent collaboration offers a promising avenue to transcend these boundaries. However, existing frameworks typically rely on prompt-based test-time interactions or multi-role configurations trained with homogeneous parameters, limiting error correction capabilities and strategic diversity. In this paper, we propose a Multi-Agent Reinforced Training and Inference Framework with Self-Search Scaling (MARTI-MARS2), which integrates policy learning with multi-agent tree search by formulating the multi-agent collaborative exploration process as a dynamic and learnable environment. By allowing agents to iteratively explore and refine within the environment, the framework facilitates evolution from parameter-sharing homogeneous multi-role training to heterogeneous multi-agent training, breaking through single-agent capability limits. We also introduce an efficient inference strategy MARTI-MARS2-T+ to fully exploit the scaling potential of multi-agent collaboration at test time. We conduct extensive experiments across varied model scales (8B, 14B, and 32B) on challenging code generation benchmarks. Utilizing two collaborating 32B models, MARTI-MARS2 achieves 77.7%, outperforming strong baselines like GPT-5.1. Furthermore, MARTI-MARS2 reveals a novel scaling law: shifting from single-agent to homogeneous multi-role and ultimately to heterogeneous multi-agent paradigms progressively yields higher RL performance ceilings, robust TTS capabilities, and greater policy diversity, suggesting that policy diversity is critical for scaling intelligence via multi-agent reinforcement learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07848 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MARTI-MARS$^2$: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation Wang, Shijie Li, Pengfei Fu, Yikun Liu, Kaifeng Li, Fangyuan Liu, Yang Sun, Xiaowei Li, Zonglin Zhao, Siyao Zhao, Jian Tian, Kai Li, Dong Gao, Junqi Zhang, Yutong Chen, Yiqun Li, Yuqiang Li, Zoe Zhang, Weinan Ye, Peng Hu, Shuyue Bai, Lei Zhou, Bowen Zhang, Kaiyan Qi, Biqing Machine Learning While the complex reasoning capability of Large Language Models (LLMs) has attracted significant attention, single-agent systems often encounter inherent performance ceilings in complex tasks such as code generation. Multi-agent collaboration offers a promising avenue to transcend these boundaries. However, existing frameworks typically rely on prompt-based test-time interactions or multi-role configurations trained with homogeneous parameters, limiting error correction capabilities and strategic diversity. In this paper, we propose a Multi-Agent Reinforced Training and Inference Framework with Self-Search Scaling (MARTI-MARS2), which integrates policy learning with multi-agent tree search by formulating the multi-agent collaborative exploration process as a dynamic and learnable environment. By allowing agents to iteratively explore and refine within the environment, the framework facilitates evolution from parameter-sharing homogeneous multi-role training to heterogeneous multi-agent training, breaking through single-agent capability limits. We also introduce an efficient inference strategy MARTI-MARS2-T+ to fully exploit the scaling potential of multi-agent collaboration at test time. We conduct extensive experiments across varied model scales (8B, 14B, and 32B) on challenging code generation benchmarks. Utilizing two collaborating 32B models, MARTI-MARS2 achieves 77.7%, outperforming strong baselines like GPT-5.1. Furthermore, MARTI-MARS2 reveals a novel scaling law: shifting from single-agent to homogeneous multi-role and ultimately to heterogeneous multi-agent paradigms progressively yields higher RL performance ceilings, robust TTS capabilities, and greater policy diversity, suggesting that policy diversity is critical for scaling intelligence via multi-agent reinforcement learning. |
| title | MARTI-MARS$^2$: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.07848 |