MARS$^2$: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914478988197888 |
|---|---|
| author | Li, Pengfei Wang, Shijie Li, Fangyuan Fu, Yikun Liu, Kaifeng Zhang, Kaiyan Zhang, Dazhi Li, Yuqiang Qi, Biqing Zhou, Bowen |
| author_facet | Li, Pengfei Wang, Shijie Li, Fangyuan Fu, Yikun Liu, Kaifeng Zhang, Kaiyan Zhang, Dazhi Li, Yuqiang Qi, Biqing Zhou, Bowen |
| contents | Reinforcement learning (RL) paradigms have demonstrated strong performance on reasoning-intensive tasks such as code generation. However, limited trajectory diversity often leads to diminishing returns, which constrains the achievable performance ceiling. Search-enhanced RL alleviates this issue by introducing structured exploration, which remains constrained by the single-agent policy priors. Meanwhile, leveraging multiple interacting policies can acquire more diverse exploratory signals, but existing approaches are typically decoupled from structured search. We propose \textbf{MARS$^2$} (Multi-Agent Reinforced Tree-Search Scaling), a unified RL framework in which multiple independently-optimized agents collaborate within a shared tree-structured search environment. MARS$^2$ models the search tree as a learnable multi-agent interaction environment, enabling heterogeneous agents to collaboratively generate and refine candidate solutions within a shared search topology. To support effective learning, we introduce a path-level group advantage formulation based on tree-consistent reward shaping, which facilitates effective credit assignment across complex search trajectories. Experiments on code generation benchmarks show that MARS$^2$ consistently improves performance across diverse model combinations and training settings, demonstrating the effectiveness of coupling multi-agent collaboration with tree search for enhancing reinforcement learning. Our code is publicly available at https://github.com/TsinghuaC3I/MARTI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_14564 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MARS$^2$: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation Li, Pengfei Wang, Shijie Li, Fangyuan Fu, Yikun Liu, Kaifeng Zhang, Kaiyan Zhang, Dazhi Li, Yuqiang Qi, Biqing Zhou, Bowen Artificial Intelligence Computation and Language Reinforcement learning (RL) paradigms have demonstrated strong performance on reasoning-intensive tasks such as code generation. However, limited trajectory diversity often leads to diminishing returns, which constrains the achievable performance ceiling. Search-enhanced RL alleviates this issue by introducing structured exploration, which remains constrained by the single-agent policy priors. Meanwhile, leveraging multiple interacting policies can acquire more diverse exploratory signals, but existing approaches are typically decoupled from structured search. We propose \textbf{MARS$^2$} (Multi-Agent Reinforced Tree-Search Scaling), a unified RL framework in which multiple independently-optimized agents collaborate within a shared tree-structured search environment. MARS$^2$ models the search tree as a learnable multi-agent interaction environment, enabling heterogeneous agents to collaboratively generate and refine candidate solutions within a shared search topology. To support effective learning, we introduce a path-level group advantage formulation based on tree-consistent reward shaping, which facilitates effective credit assignment across complex search trajectories. Experiments on code generation benchmarks show that MARS$^2$ consistently improves performance across diverse model combinations and training settings, demonstrating the effectiveness of coupling multi-agent collaboration with tree search for enhancing reinforcement learning. Our code is publicly available at https://github.com/TsinghuaC3I/MARTI. |
| title | MARS$^2$: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2604.14564 |