LTL-Constrained Policy Optimization with Cycle Experience Replay
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916661136719872 |
|---|---|
| author | Shah, Ameesh Voloshin, Cameron Yang, Chenxi Verma, Abhinav Chaudhuri, Swarat Seshia, Sanjit A. |
| author_facet | Shah, Ameesh Voloshin, Cameron Yang, Chenxi Verma, Abhinav Chaudhuri, Swarat Seshia, Sanjit A. |
| contents | Linear Temporal Logic (LTL) offers a precise means for constraining the behavior of reinforcement learning agents. However, in many settings where both satisfaction and optimality conditions are present, LTL is insufficient to capture both. Instead, LTL-constrained policy optimization, where the goal is to optimize a scalar reward under LTL constraints, is needed. This constrained optimization problem proves difficult in deep Reinforcement Learning (DRL) settings, where learned policies often ignore the LTL constraint due to the sparse nature of LTL satisfaction. To alleviate the sparsity issue, we introduce Cycle Experience Replay (CyclER), a novel reward shaping technique that exploits the underlying structure of the LTL constraint to guide a policy towards satisfaction by encouraging partial behaviors compliant with the constraint. We provide a theoretical guarantee that optimizing CyclER will achieve policies that satisfy the LTL constraint with near-optimal probability. We evaluate CyclER in three continuous control domains. Our experimental results show that optimizing CyclER in tandem with the existing scalar reward outperforms existing reward-shaping methods at finding performant LTL-satisfying policies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_11578 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LTL-Constrained Policy Optimization with Cycle Experience Replay Shah, Ameesh Voloshin, Cameron Yang, Chenxi Verma, Abhinav Chaudhuri, Swarat Seshia, Sanjit A. Machine Learning Artificial Intelligence Formal Languages and Automata Theory I.2.6; I.2.4 Linear Temporal Logic (LTL) offers a precise means for constraining the behavior of reinforcement learning agents. However, in many settings where both satisfaction and optimality conditions are present, LTL is insufficient to capture both. Instead, LTL-constrained policy optimization, where the goal is to optimize a scalar reward under LTL constraints, is needed. This constrained optimization problem proves difficult in deep Reinforcement Learning (DRL) settings, where learned policies often ignore the LTL constraint due to the sparse nature of LTL satisfaction. To alleviate the sparsity issue, we introduce Cycle Experience Replay (CyclER), a novel reward shaping technique that exploits the underlying structure of the LTL constraint to guide a policy towards satisfaction by encouraging partial behaviors compliant with the constraint. We provide a theoretical guarantee that optimizing CyclER will achieve policies that satisfy the LTL constraint with near-optimal probability. We evaluate CyclER in three continuous control domains. Our experimental results show that optimizing CyclER in tandem with the existing scalar reward outperforms existing reward-shaping methods at finding performant LTL-satisfying policies. |
| title | LTL-Constrained Policy Optimization with Cycle Experience Replay |
| topic | Machine Learning Artificial Intelligence Formal Languages and Automata Theory I.2.6; I.2.4 |
| url | https://arxiv.org/abs/2404.11578 |