Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914126852259840 |
|---|---|
| author | Levina, Kristina Pappas, Nikolaos Karapantelakis, Athanasios Feljan, Aneta Vulgarakis Seipp, Jendrik |
| author_facet | Levina, Kristina Pappas, Nikolaos Karapantelakis, Athanasios Feljan, Aneta Vulgarakis Seipp, Jendrik |
| contents | Reward machines (RMs) inform reinforcement learning agents about the reward structure of the environment. This is particularly advantageous for complex non-Markovian tasks because agents with access to RMs can learn more efficiently from fewer samples. However, learning with RMs is ill-suited for long-horizon problems in which a set of subtasks can be executed in any order. In such cases, the amount of information to learn increases exponentially with the number of unordered subtasks. In this work, we address this limitation by introducing three generalisations of RMs: (1) Numeric RMs allow users to express complex tasks in a compact form. (2) In Agenda RMs, states are associated with an agenda that tracks the remaining subtasks to complete. (3) Coupled RMs have coupled states associated with each subtask in the agenda. Furthermore, we introduce a new compositional learning algorithm that leverages coupled RMs: Q-learning with coupled RMs (CoRM). Our experiments show that CoRM scales better than state-of-the-art RM algorithms for long-horizon problems with unordered subtasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_27329 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines Levina, Kristina Pappas, Nikolaos Karapantelakis, Athanasios Feljan, Aneta Vulgarakis Seipp, Jendrik Artificial Intelligence Reward machines (RMs) inform reinforcement learning agents about the reward structure of the environment. This is particularly advantageous for complex non-Markovian tasks because agents with access to RMs can learn more efficiently from fewer samples. However, learning with RMs is ill-suited for long-horizon problems in which a set of subtasks can be executed in any order. In such cases, the amount of information to learn increases exponentially with the number of unordered subtasks. In this work, we address this limitation by introducing three generalisations of RMs: (1) Numeric RMs allow users to express complex tasks in a compact form. (2) In Agenda RMs, states are associated with an agenda that tracks the remaining subtasks to complete. (3) Coupled RMs have coupled states associated with each subtask in the agenda. Furthermore, we introduce a new compositional learning algorithm that leverages coupled RMs: Q-learning with coupled RMs (CoRM). Our experiments show that CoRM scales better than state-of-the-art RM algorithms for long-horizon problems with unordered subtasks. |
| title | Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.27329 |