Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917088225918976 |
|---|---|
| author | Hong, Haoyang Yin, Jiajun Wang, Yuan Liu, Jingnan Chen, Zhe Yu, Ailing Li, Ji Ye, Zhiling Xiao, Hansong Chen, Yefei Zhou, Hualei Yue, Yun Yang, Minghui Guo, Chunxiao Liu, Junwei Wei, Peng Gu, Jinjie |
| author_facet | Hong, Haoyang Yin, Jiajun Wang, Yuan Liu, Jingnan Chen, Zhe Yu, Ailing Li, Ji Ye, Zhiling Xiao, Hansong Chen, Yefei Zhou, Hualei Yue, Yun Yang, Minghui Guo, Chunxiao Liu, Junwei Wei, Peng Gu, Jinjie |
| contents | Multi-agent systems perform well on general reasoning tasks. However, the lack of training in specialized areas hinders their accuracy. Current training methods train a unified large language model (LLM) for all agents in the system. This may limit the performances due to different distributions underlying for different agents. Therefore, training multi-agent systems with distinct LLMs should be the next step to solve. However, this approach introduces optimization challenges. For example, agents operate at different frequencies, rollouts involve varying sub-agent invocations, and agents are often deployed across separate servers, disrupting end-to-end gradient flow. To address these issues, we propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization designed for vertical Multi-agent systems with a main agent (planner) and multiple sub-agents (multi-turn tool executors). M-GRPO computes group-relative advantages for both main and sub-agents, maintaining hierarchical credit assignment. It also introduces a trajectory-alignment scheme that generates fixed-size batches despite variable sub-agent invocations. We deploy a decoupled training pipeline in which agents run on separate servers and exchange minimal statistics via a shared store. This enables scalable training without cross-server backpropagation. In experiments on real-world benchmarks (e.g., GAIA, XBench-DeepSearch, and WebWalkerQA), M-GRPO consistently outperforms both single-agent GRPO and multi-agent GRPO with frozen sub-agents, demonstrating improved stability and sample efficiency. These results show that aligning heterogeneous trajectories and decoupling optimization across specialized agents enhances tool-augmented reasoning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_13288 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO Hong, Haoyang Yin, Jiajun Wang, Yuan Liu, Jingnan Chen, Zhe Yu, Ailing Li, Ji Ye, Zhiling Xiao, Hansong Chen, Yefei Zhou, Hualei Yue, Yun Yang, Minghui Guo, Chunxiao Liu, Junwei Wei, Peng Gu, Jinjie Artificial Intelligence Multi-agent systems perform well on general reasoning tasks. However, the lack of training in specialized areas hinders their accuracy. Current training methods train a unified large language model (LLM) for all agents in the system. This may limit the performances due to different distributions underlying for different agents. Therefore, training multi-agent systems with distinct LLMs should be the next step to solve. However, this approach introduces optimization challenges. For example, agents operate at different frequencies, rollouts involve varying sub-agent invocations, and agents are often deployed across separate servers, disrupting end-to-end gradient flow. To address these issues, we propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization designed for vertical Multi-agent systems with a main agent (planner) and multiple sub-agents (multi-turn tool executors). M-GRPO computes group-relative advantages for both main and sub-agents, maintaining hierarchical credit assignment. It also introduces a trajectory-alignment scheme that generates fixed-size batches despite variable sub-agent invocations. We deploy a decoupled training pipeline in which agents run on separate servers and exchange minimal statistics via a shared store. This enables scalable training without cross-server backpropagation. In experiments on real-world benchmarks (e.g., GAIA, XBench-DeepSearch, and WebWalkerQA), M-GRPO consistently outperforms both single-agent GRPO and multi-agent GRPO with frozen sub-agents, demonstrating improved stability and sample efficiency. These results show that aligning heterogeneous trajectories and decoupling optimization across specialized agents enhances tool-augmented reasoning tasks. |
| title | Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2511.13288 |