Constraint-Aware Reinforcement Learning via Adaptive Action Scaling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914546840502272 |
|---|---|
| author | Dawood, Murad Siddiquie, Usama Ahmed Khorshidi, Shahram Bennewitz, Maren |
| author_facet | Dawood, Murad Siddiquie, Usama Ahmed Khorshidi, Shahram Bennewitz, Maren |
| contents | Safe reinforcement learning (RL) seeks to mitigate unsafe behaviors that arise from exploration during training by reducing constraint violations while maintaining task performance. Existing approaches typically rely on a single policy to jointly optimize reward and safety, which can cause instability due to conflicting objectives, or they use external safety filters that override actions and require prior system knowledge. In this paper, we propose a modular cost-aware regulator that scales the agent's actions based on predicted constraint violations, preserving exploration through smooth action modulation rather than overriding the policy. The regulator is trained to minimize constraint violations while avoiding degenerate suppression of actions. Our approach integrates seamlessly with off-policy RL methods such as SAC and TD3, and achieves state-of-the-art return-to-cost ratios on Safety Gym locomotion tasks with sparse costs, reducing constraint violations by up to 126 times while increasing returns by over an order of magnitude compared to prior methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_11491 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Constraint-Aware Reinforcement Learning via Adaptive Action Scaling Dawood, Murad Siddiquie, Usama Ahmed Khorshidi, Shahram Bennewitz, Maren Robotics Machine Learning Systems and Control Safe reinforcement learning (RL) seeks to mitigate unsafe behaviors that arise from exploration during training by reducing constraint violations while maintaining task performance. Existing approaches typically rely on a single policy to jointly optimize reward and safety, which can cause instability due to conflicting objectives, or they use external safety filters that override actions and require prior system knowledge. In this paper, we propose a modular cost-aware regulator that scales the agent's actions based on predicted constraint violations, preserving exploration through smooth action modulation rather than overriding the policy. The regulator is trained to minimize constraint violations while avoiding degenerate suppression of actions. Our approach integrates seamlessly with off-policy RL methods such as SAC and TD3, and achieves state-of-the-art return-to-cost ratios on Safety Gym locomotion tasks with sparse costs, reducing constraint violations by up to 126 times while increasing returns by over an order of magnitude compared to prior methods. |
| title | Constraint-Aware Reinforcement Learning via Adaptive Action Scaling |
| topic | Robotics Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2510.11491 |