Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.08417 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918052612800512 |
|---|---|
| author | Yao, Qingmao Lei, Zhichao Chen, Tianyuan Yuan, Ziyue Chen, Xuefan Liu, Jianxiang Wu, Faguo Zhang, Xiao |
| author_facet | Yao, Qingmao Lei, Zhichao Chen, Tianyuan Yuan, Ziyue Chen, Xuefan Liu, Jianxiang Wu, Faguo Zhang, Xiao |
| contents | Offline Reinforcement Learning (RL) struggles with distributional shifts, leading to the $Q$-value overestimation for out-of-distribution (OOD) actions. Existing methods address this issue by imposing constraints; however, they often become overly conservative when evaluating OOD regions, which constrains the $Q$-function generalization. This over-constraint issue results in poor $Q$-value estimation and hinders policy improvement. In this paper, we introduce a novel approach to achieve better $Q$-value estimation by enhancing $Q$-function generalization in OOD regions within Convex Hull and its Neighborhood (CHN). Under the safety generalization guarantees of the CHN, we propose the Smooth Bellman Operator (SBO), which updates OOD $Q$-values by smoothing them with neighboring in-sample $Q$-values. We theoretically show that SBO approximates true $Q$-values for both in-sample and OOD actions within the CHN. Our practical algorithm, Smooth Q-function OOD Generalization (SQOG), empirically alleviates the over-constraint issue, achieving near-accurate $Q$-value estimation. On the D4RL benchmarks, SQOG outperforms existing state-of-the-art methods in both performance and computational efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08417 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood Yao, Qingmao Lei, Zhichao Chen, Tianyuan Yuan, Ziyue Chen, Xuefan Liu, Jianxiang Wu, Faguo Zhang, Xiao Machine Learning Artificial Intelligence Offline Reinforcement Learning (RL) struggles with distributional shifts, leading to the $Q$-value overestimation for out-of-distribution (OOD) actions. Existing methods address this issue by imposing constraints; however, they often become overly conservative when evaluating OOD regions, which constrains the $Q$-function generalization. This over-constraint issue results in poor $Q$-value estimation and hinders policy improvement. In this paper, we introduce a novel approach to achieve better $Q$-value estimation by enhancing $Q$-function generalization in OOD regions within Convex Hull and its Neighborhood (CHN). Under the safety generalization guarantees of the CHN, we propose the Smooth Bellman Operator (SBO), which updates OOD $Q$-values by smoothing them with neighboring in-sample $Q$-values. We theoretically show that SBO approximates true $Q$-values for both in-sample and OOD actions within the CHN. Our practical algorithm, Smooth Q-function OOD Generalization (SQOG), empirically alleviates the over-constraint issue, achieving near-accurate $Q$-value estimation. On the D4RL benchmarks, SQOG outperforms existing state-of-the-art methods in both performance and computational efficiency. |
| title | Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.08417 |