Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910753860091904 |
|---|---|
| author | Bejarano, Federico Pizarro Brunke, Lukas Schoellig, Angela P. |
| author_facet | Bejarano, Federico Pizarro Brunke, Lukas Schoellig, Angela P. |
| contents | Reinforcement learning (RL) controllers are flexible and performant but rarely guarantee safety. Safety filters impart hard safety guarantees to RL controllers while maintaining flexibility. However, safety filters can cause undesired behaviours due to the separation between the controller and the safety filter, often degrading performance and robustness. In this paper, we analyze several modifications to incorporating the safety filter in training RL controllers rather than solely applying it during evaluation. The modifications allow the RL controller to learn to account for the safety filter, improving performance. This paper presents a comprehensive analysis of training RL with safety filters, featuring simulated and real-world experiments with a Crazyflie 2.0 drone. We examine how various training modifications and hyperparameters impact performance, sample efficiency, safety, and chattering. Our findings serve as a guide for practitioners and researchers focused on safety filters and safe RL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11671 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents Bejarano, Federico Pizarro Brunke, Lukas Schoellig, Angela P. Robotics Machine Learning Systems and Control Reinforcement learning (RL) controllers are flexible and performant but rarely guarantee safety. Safety filters impart hard safety guarantees to RL controllers while maintaining flexibility. However, safety filters can cause undesired behaviours due to the separation between the controller and the safety filter, often degrading performance and robustness. In this paper, we analyze several modifications to incorporating the safety filter in training RL controllers rather than solely applying it during evaluation. The modifications allow the RL controller to learn to account for the safety filter, improving performance. This paper presents a comprehensive analysis of training RL with safety filters, featuring simulated and real-world experiments with a Crazyflie 2.0 drone. We examine how various training modifications and hyperparameters impact performance, sample efficiency, safety, and chattering. Our findings serve as a guide for practitioners and researchers focused on safety filters and safe RL. |
| title | Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents |
| topic | Robotics Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2410.11671 |