Counting Reward Automata: Sample Efficient Reinforcement Learning Through the Exploitation of Reward Function Structure
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913236118405120 |
|---|---|
| author | Bester, Tristan Rosman, Benjamin James, Steven Tasse, Geraud Nangue |
| author_facet | Bester, Tristan Rosman, Benjamin James, Steven Tasse, Geraud Nangue |
| contents | We present counting reward automata-a finite state machine variant capable of modelling any reward function expressible as a formal language. Unlike previous approaches, which are limited to the expression of tasks as regular languages, our framework allows for tasks described by unrestricted grammars. We prove that an agent equipped with such an abstract machine is able to solve a larger set of tasks than those utilising current approaches. We show that this increase in expressive power does not come at the cost of increased automaton complexity. A selection of learning algorithms are presented which exploit automaton structure to improve sample efficiency. We show that the state machines required in our formulation can be specified from natural language task descriptions using large language models. Empirical results demonstrate that our method outperforms competing approaches in terms of sample efficiency, automaton complexity, and task completion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_11364 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Counting Reward Automata: Sample Efficient Reinforcement Learning Through the Exploitation of Reward Function Structure Bester, Tristan Rosman, Benjamin James, Steven Tasse, Geraud Nangue Artificial Intelligence I.2; F.4 We present counting reward automata-a finite state machine variant capable of modelling any reward function expressible as a formal language. Unlike previous approaches, which are limited to the expression of tasks as regular languages, our framework allows for tasks described by unrestricted grammars. We prove that an agent equipped with such an abstract machine is able to solve a larger set of tasks than those utilising current approaches. We show that this increase in expressive power does not come at the cost of increased automaton complexity. A selection of learning algorithms are presented which exploit automaton structure to improve sample efficiency. We show that the state machines required in our formulation can be specified from natural language task descriptions using large language models. Empirical results demonstrate that our method outperforms competing approaches in terms of sample efficiency, automaton complexity, and task completion. |
| title | Counting Reward Automata: Sample Efficient Reinforcement Learning Through the Exploitation of Reward Function Structure |
| topic | Artificial Intelligence I.2; F.4 |
| url | https://arxiv.org/abs/2312.11364 |