Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.13327 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910509205291008 |
|---|---|
| author | Sinii, Viacheslav Nikulin, Alexander Kurenkov, Vladislav Zisman, Ilya Kolesnikov, Sergey |
| author_facet | Sinii, Viacheslav Nikulin, Alexander Kurenkov, Vladislav Zisman, Ilya Kolesnikov, Sergey |
| contents | Recently, it has been shown that transformers pre-trained on diverse datasets with multi-episode contexts can generalize to new reinforcement learning tasks in-context. A key limitation of previously proposed models is their reliance on a predefined action space size and structure. The introduction of a new action space often requires data re-collection and model re-training, which can be costly for some applications. In our work, we show that it is possible to mitigate this issue by proposing the Headless-AD model that, despite being trained only once, is capable of generalizing to discrete action spaces of variable size, semantic content and order. By experimenting with Bernoulli and contextual bandits, as well as a gridworld environment, we show that Headless-AD exhibits significant capability to generalize to action spaces it has never encountered, even outperforming specialized models trained for a specific set of actions on several environment configurations. Implementation is available at: https://github.com/corl-team/headless-ad. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_13327 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | In-Context Reinforcement Learning for Variable Action Spaces Sinii, Viacheslav Nikulin, Alexander Kurenkov, Vladislav Zisman, Ilya Kolesnikov, Sergey Machine Learning Artificial Intelligence Recently, it has been shown that transformers pre-trained on diverse datasets with multi-episode contexts can generalize to new reinforcement learning tasks in-context. A key limitation of previously proposed models is their reliance on a predefined action space size and structure. The introduction of a new action space often requires data re-collection and model re-training, which can be costly for some applications. In our work, we show that it is possible to mitigate this issue by proposing the Headless-AD model that, despite being trained only once, is capable of generalizing to discrete action spaces of variable size, semantic content and order. By experimenting with Bernoulli and contextual bandits, as well as a gridworld environment, we show that Headless-AD exhibits significant capability to generalize to action spaces it has never encountered, even outperforming specialized models trained for a specific set of actions on several environment configurations. Implementation is available at: https://github.com/corl-team/headless-ad. |
| title | In-Context Reinforcement Learning for Variable Action Spaces |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2312.13327 |