Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916232360361984 |
|---|---|
| author | Rietz, Finn Schaffernicht, Erik Heinrich, Stefan Stork, Johannes A. |
| author_facet | Rietz, Finn Schaffernicht, Erik Heinrich, Stefan Stork, Johannes A. |
| contents | Reinforcement learning policies are typically represented by black-box neural networks, which are non-interpretable and not well-suited for safety-critical domains. To address both of these issues, we propose constrained normalizing flow policies as interpretable and safe-by-construction policy models. We achieve safety for reinforcement learning problems with instantaneous safety constraints, for which we can exploit domain knowledge by analytically constructing a normalizing flow that ensures constraint satisfaction. The normalizing flow corresponds to an interpretable sequence of transformations on action samples, each ensuring alignment with respect to a particular constraint. Our experiments reveal benefits beyond interpretability in an easier learning objective and maintained constraint satisfaction throughout the entire learning process. Our approach leverages constraints over reward engineering while offering enhanced interpretability, safety, and direct means of providing domain knowledge to the agent without relying on complex reward functions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_01198 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies Rietz, Finn Schaffernicht, Erik Heinrich, Stefan Stork, Johannes A. Machine Learning Artificial Intelligence Reinforcement learning policies are typically represented by black-box neural networks, which are non-interpretable and not well-suited for safety-critical domains. To address both of these issues, we propose constrained normalizing flow policies as interpretable and safe-by-construction policy models. We achieve safety for reinforcement learning problems with instantaneous safety constraints, for which we can exploit domain knowledge by analytically constructing a normalizing flow that ensures constraint satisfaction. The normalizing flow corresponds to an interpretable sequence of transformations on action samples, each ensuring alignment with respect to a particular constraint. Our experiments reveal benefits beyond interpretability in an easier learning objective and maintained constraint satisfaction throughout the entire learning process. Our approach leverages constraints over reward engineering while offering enhanced interpretability, safety, and direct means of providing domain knowledge to the agent without relying on complex reward functions. |
| title | Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.01198 |