Selective Reincarnation: Offline-to-Online Multi-Agent Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910677910683648 |
|---|---|
| author | Formanek, Claude Tilbury, Callum Rhys Shock, Jonathan Tessera, Kale-ab Pretorius, Arnu |
| author_facet | Formanek, Claude Tilbury, Callum Rhys Shock, Jonathan Tessera, Kale-ab Pretorius, Arnu |
| contents | 'Reincarnation' in reinforcement learning has been proposed as a formalisation of reusing prior computation from past experiments when training an agent in an environment. In this paper, we present a brief foray into the paradigm of reincarnation in the multi-agent (MA) context. We consider the case where only some agents are reincarnated, whereas the others are trained from scratch -- selective reincarnation. In the fully-cooperative MA setting with heterogeneous agents, we demonstrate that selective reincarnation can lead to higher returns than training fully from scratch, and faster convergence than training with full reincarnation. However, the choice of which agents to reincarnate in a heterogeneous system is vitally important to the outcome of the training -- in fact, a poor choice can lead to considerably worse results than the alternatives. We argue that a rich field of work exists here, and we hope that our effort catalyses further energy in bringing the topic of reincarnation to the multi-agent realm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_00977 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Selective Reincarnation: Offline-to-Online Multi-Agent Reinforcement Learning Formanek, Claude Tilbury, Callum Rhys Shock, Jonathan Tessera, Kale-ab Pretorius, Arnu Artificial Intelligence Machine Learning Multiagent Systems 'Reincarnation' in reinforcement learning has been proposed as a formalisation of reusing prior computation from past experiments when training an agent in an environment. In this paper, we present a brief foray into the paradigm of reincarnation in the multi-agent (MA) context. We consider the case where only some agents are reincarnated, whereas the others are trained from scratch -- selective reincarnation. In the fully-cooperative MA setting with heterogeneous agents, we demonstrate that selective reincarnation can lead to higher returns than training fully from scratch, and faster convergence than training with full reincarnation. However, the choice of which agents to reincarnate in a heterogeneous system is vitally important to the outcome of the training -- in fact, a poor choice can lead to considerably worse results than the alternatives. We argue that a rich field of work exists here, and we hope that our effort catalyses further energy in bringing the topic of reincarnation to the multi-agent realm. |
| title | Selective Reincarnation: Offline-to-Online Multi-Agent Reinforcement Learning |
| topic | Artificial Intelligence Machine Learning Multiagent Systems |
| url | https://arxiv.org/abs/2304.00977 |