Action Mapping for Reinforcement Learning in Continuous Environments with Constraints
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913599253905408 |
|---|---|
| author | Theile, Mirco Dirnberger, Lukas Trumpp, Raphael Caccamo, Marco Sangiovanni-Vincentelli, Alberto L. |
| author_facet | Theile, Mirco Dirnberger, Lukas Trumpp, Raphael Caccamo, Marco Sangiovanni-Vincentelli, Alberto L. |
| contents | Deep reinforcement learning (DRL) has had success across various domains, but applying it to environments with constraints remains challenging due to poor sample efficiency and slow convergence. Recent literature explored incorporating model knowledge to mitigate these problems, particularly through the use of models that assess the feasibility of proposed actions. However, integrating feasibility models efficiently into DRL pipelines in environments with continuous action spaces is non-trivial. We propose a novel DRL training strategy utilizing action mapping that leverages feasibility models to streamline the learning process. By decoupling the learning of feasible actions from policy optimization, action mapping allows DRL agents to focus on selecting the optimal action from a reduced feasible action set. We demonstrate through experiments that action mapping significantly improves training performance in constrained environments with continuous action spaces, especially with imperfect feasibility models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04327 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Action Mapping for Reinforcement Learning in Continuous Environments with Constraints Theile, Mirco Dirnberger, Lukas Trumpp, Raphael Caccamo, Marco Sangiovanni-Vincentelli, Alberto L. Machine Learning Artificial Intelligence Systems and Control Deep reinforcement learning (DRL) has had success across various domains, but applying it to environments with constraints remains challenging due to poor sample efficiency and slow convergence. Recent literature explored incorporating model knowledge to mitigate these problems, particularly through the use of models that assess the feasibility of proposed actions. However, integrating feasibility models efficiently into DRL pipelines in environments with continuous action spaces is non-trivial. We propose a novel DRL training strategy utilizing action mapping that leverages feasibility models to streamline the learning process. By decoupling the learning of feasible actions from policy optimization, action mapping allows DRL agents to focus on selecting the optimal action from a reduced feasible action set. We demonstrate through experiments that action mapping significantly improves training performance in constrained environments with continuous action spaces, especially with imperfect feasibility models. |
| title | Action Mapping for Reinforcement Learning in Continuous Environments with Constraints |
| topic | Machine Learning Artificial Intelligence Systems and Control |
| url | https://arxiv.org/abs/2412.04327 |