Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910044762669056 |
|---|---|
| author | Kim, Seongmin Park, Giseung Kim, Woojun Jeon, Jiwon Han, Seungyul Sung, Youngchul |
| author_facet | Kim, Seongmin Park, Giseung Kim, Woojun Jeon, Jiwon Han, Seungyul Sung, Youngchul |
| contents | In this paper, we propose a novel framework for multi-agent reinforcement learning that enhances sample efficiency and coordination through accurate per-agent advantage estimation. The core of our approach is Generalized Per-Agent Advantage Estimator (GPAE), which employs a per-agent value iteration operator to compute precise per-agent advantages. This operator enables stable off-policy learning by indirectly estimating values via action probabilities, eliminating the need for direct Q-function estimation. To further refine estimation, we introduce a double-truncated importance sampling ratio scheme. This scheme improves credit assignment for off-policy trajectories by balancing sensitivity to the agent's own policy changes with robustness to non-stationarity from other agents. Experiments on benchmarks demonstrate that our approach outperforms existing approaches, excelling in coordination and sample efficiency for complex scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02654 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization Kim, Seongmin Park, Giseung Kim, Woojun Jeon, Jiwon Han, Seungyul Sung, Youngchul Multiagent Systems In this paper, we propose a novel framework for multi-agent reinforcement learning that enhances sample efficiency and coordination through accurate per-agent advantage estimation. The core of our approach is Generalized Per-Agent Advantage Estimator (GPAE), which employs a per-agent value iteration operator to compute precise per-agent advantages. This operator enables stable off-policy learning by indirectly estimating values via action probabilities, eliminating the need for direct Q-function estimation. To further refine estimation, we introduce a double-truncated importance sampling ratio scheme. This scheme improves credit assignment for off-policy trajectories by balancing sensitivity to the agent's own policy changes with robustness to non-stationarity from other agents. Experiments on benchmarks demonstrate that our approach outperforms existing approaches, excelling in coordination and sample efficiency for complex scenarios. |
| title | Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization |
| topic | Multiagent Systems |
| url | https://arxiv.org/abs/2603.02654 |