MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912899848470528 |
|---|---|
| author | Yuan, Huining Xu, Zelai Tan, Zheyue Yi, Xiangmin Guang, Mo Long, Kaiwen Hui, Haojia Li, Boxun Chen, Xinlei Zhao, Bo Zhang, Xiao-Ping Yu, Chao Wang, Yu |
| author_facet | Yuan, Huining Xu, Zelai Tan, Zheyue Yi, Xiangmin Guang, Mo Long, Kaiwen Hui, Haojia Li, Boxun Chen, Xinlei Zhao, Bo Zhang, Xiao-Ping Yu, Chao Wang, Yu |
| contents | Developing Large Language Models (LLMs) to cooperate and compete effectively within multi-agent systems (MASs) is a critical step towards more advanced intelligence. While reinforcement learning (RL) has proven effective for enhancing reasoning in single-agent tasks, its extension to multi-turn, multi-agent scenarios remains underexplored due to the challenges of long-horizon credit assignment and agent-specific advantage estimation. To address these challenges, we introduce MARSHAL, an end-to-end RL framework that incentivizes Multi-Agent Reasoning through Self-play witH strAtegic LLMs in both cooperative and competitive games. MARSHAL features a turn-level advantage estimator that aligns learning signals with each interaction for credit assignment, and an agent-specific advantage normalization to stabilize multi-agent training. By learning with self-play across cooperative and competitive games, MARSHAL agents trained from Qwen3-4B develop strong strategic abilities, with up to 28.7% performance improvements in held-out games. More importantly, the capability acquired through self-play generalizes beyond games, yielding consistent performance gains of MASs in reasoning benchmarks. When integrated into leading MASs, our MARSHAL agent achieves significant zero-shot performance gains of up to 10.0% on AIME, 7.6% on GPQA-Diamond, and 3.5% on average across all benchmarks. These results establish self-play in strategic games as a powerful approach for developing generalizable multi-agent reasoning capabilities in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_15414 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs Yuan, Huining Xu, Zelai Tan, Zheyue Yi, Xiangmin Guang, Mo Long, Kaiwen Hui, Haojia Li, Boxun Chen, Xinlei Zhao, Bo Zhang, Xiao-Ping Yu, Chao Wang, Yu Artificial Intelligence Developing Large Language Models (LLMs) to cooperate and compete effectively within multi-agent systems (MASs) is a critical step towards more advanced intelligence. While reinforcement learning (RL) has proven effective for enhancing reasoning in single-agent tasks, its extension to multi-turn, multi-agent scenarios remains underexplored due to the challenges of long-horizon credit assignment and agent-specific advantage estimation. To address these challenges, we introduce MARSHAL, an end-to-end RL framework that incentivizes Multi-Agent Reasoning through Self-play witH strAtegic LLMs in both cooperative and competitive games. MARSHAL features a turn-level advantage estimator that aligns learning signals with each interaction for credit assignment, and an agent-specific advantage normalization to stabilize multi-agent training. By learning with self-play across cooperative and competitive games, MARSHAL agents trained from Qwen3-4B develop strong strategic abilities, with up to 28.7% performance improvements in held-out games. More importantly, the capability acquired through self-play generalizes beyond games, yielding consistent performance gains of MASs in reasoning benchmarks. When integrated into leading MASs, our MARSHAL agent achieves significant zero-shot performance gains of up to 10.0% on AIME, 7.6% on GPQA-Diamond, and 3.5% on average across all benchmarks. These results establish self-play in strategic games as a powerful approach for developing generalizable multi-agent reasoning capabilities in LLMs. |
| title | MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.15414 |