Heterogeneous Agent Collaborative Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914586643398656 |
|---|---|
| author | Zhang, Zhixia Huang, Zixuan Li, Gongxun Wang, Huaiyang Yuan, Chengyi Xia, Xin Wang, Deqing Zhuang, Fuzhen Ma, Shuai Ding, Ning Yang, Yaodong Li, Jianxin Ban, Yikun |
| author_facet | Zhang, Zhixia Huang, Zixuan Li, Gongxun Wang, Huaiyang Yuan, Chengyi Xia, Xin Wang, Deqing Zhuang, Fuzhen Ma, Shuai Ding, Ning Yang, Yaodong Li, Jianxin Ban, Yikun |
| contents | We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02604 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Heterogeneous Agent Collaborative Reinforcement Learning Zhang, Zhixia Huang, Zixuan Li, Gongxun Wang, Huaiyang Yuan, Chengyi Xia, Xin Wang, Deqing Zhuang, Fuzhen Ma, Shuai Ding, Ning Yang, Yaodong Li, Jianxin Ban, Yikun Machine Learning We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost. |
| title | Heterogeneous Agent Collaborative Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2603.02604 |