Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913071926083584 |
|---|---|
| author | Jing, Tan Li, Xiaorui Yao, Chao Ban, Xiaojuan Fang, Yuetong Xu, Renjing Yuan, Zhaolin |
| author_facet | Jing, Tan Li, Xiaorui Yao, Chao Ban, Xiaojuan Fang, Yuetong Xu, Renjing Yuan, Zhaolin |
| contents | Offline reinforcement learning (RL) enables learning effective policies from fixed datasets without any environment interaction. Existing methods typically employ policy constraints to mitigate the distribution shift encountered during offline RL training. However, because the scale of the constraints varies across tasks and datasets of differing quality, existing methods must meticulously tune hyperparameters to match each dataset, which is time-consuming and often impractical. We propose Adaptive Scaling of Policy Constraints (ASPC), a second-order differentiable framework that dynamically balances RL and behavior cloning (BC) during training. We theoretically analyze its performance improvement guarantee. In experiments on 39 datasets across four D4RL domains, ASPC using a single hyperparameter configuration outperforms other adaptive constraint methods and state-of-the-art offline RL algorithms that require per-dataset tuning while incurring only minimal computational overhead. The code will be released at https://github.com/Colin-Jing/ASPC. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_19900 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning Jing, Tan Li, Xiaorui Yao, Chao Ban, Xiaojuan Fang, Yuetong Xu, Renjing Yuan, Zhaolin Machine Learning Offline reinforcement learning (RL) enables learning effective policies from fixed datasets without any environment interaction. Existing methods typically employ policy constraints to mitigate the distribution shift encountered during offline RL training. However, because the scale of the constraints varies across tasks and datasets of differing quality, existing methods must meticulously tune hyperparameters to match each dataset, which is time-consuming and often impractical. We propose Adaptive Scaling of Policy Constraints (ASPC), a second-order differentiable framework that dynamically balances RL and behavior cloning (BC) during training. We theoretically analyze its performance improvement guarantee. In experiments on 39 datasets across four D4RL domains, ASPC using a single hyperparameter configuration outperforms other adaptive constraint methods and state-of-the-art offline RL algorithms that require per-dataset tuning while incurring only minimal computational overhead. The code will be released at https://github.com/Colin-Jing/ASPC. |
| title | Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.19900 |