Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912188897165312 |
|---|---|
| author | Yalcinkaya, Beyazit Lauffer, Niklas Vazquez-Chanlatte, Marcell Seshia, Sanjit A. |
| author_facet | Yalcinkaya, Beyazit Lauffer, Niklas Vazquez-Chanlatte, Marcell Seshia, Sanjit A. |
| contents | Goal-conditioned reinforcement learning is a powerful way to control an AI agent's behavior at runtime. That said, popular goal representations, e.g., target states or natural language, are either limited to Markovian tasks or rely on ambiguous task semantics. We propose representing temporal goals using compositions of deterministic finite automata (cDFAs) and use cDFAs to guide RL agents. cDFAs balance the need for formal temporal semantics with ease of interpretation: if one can understand a flow chart, one can understand a cDFA. On the other hand, cDFAs form a countably infinite concept class with Boolean semantics, and subtle changes to the automaton can result in very different tasks, making them difficult to condition agent behavior on. To address this, we observe that all paths through a DFA correspond to a series of reach-avoid tasks and propose pre-training graph neural network embeddings on "reach-avoid derived" DFAs. Through empirical evaluation, we demonstrate that the proposed pre-training method enables zero-shot generalization to various cDFA task classes and accelerated policy specialization without the myopic suboptimality of hierarchical methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_00205 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning Yalcinkaya, Beyazit Lauffer, Niklas Vazquez-Chanlatte, Marcell Seshia, Sanjit A. Machine Learning Artificial Intelligence Computation and Language Formal Languages and Automata Theory Goal-conditioned reinforcement learning is a powerful way to control an AI agent's behavior at runtime. That said, popular goal representations, e.g., target states or natural language, are either limited to Markovian tasks or rely on ambiguous task semantics. We propose representing temporal goals using compositions of deterministic finite automata (cDFAs) and use cDFAs to guide RL agents. cDFAs balance the need for formal temporal semantics with ease of interpretation: if one can understand a flow chart, one can understand a cDFA. On the other hand, cDFAs form a countably infinite concept class with Boolean semantics, and subtle changes to the automaton can result in very different tasks, making them difficult to condition agent behavior on. To address this, we observe that all paths through a DFA correspond to a series of reach-avoid tasks and propose pre-training graph neural network embeddings on "reach-avoid derived" DFAs. Through empirical evaluation, we demonstrate that the proposed pre-training method enables zero-shot generalization to various cDFA task classes and accelerated policy specialization without the myopic suboptimality of hierarchical methods. |
| title | Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning |
| topic | Machine Learning Artificial Intelligence Computation and Language Formal Languages and Automata Theory |
| url | https://arxiv.org/abs/2411.00205 |