Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2310.17245 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910123506532352 |
|---|---|
| author | Li, Hao Zhou, Xiao-Hu Li, Shu-Hai Gui, Mei-Jiang Xie, Xiao-Liang Liu, Shi-Qi Wang, Shuang-Yi Feng, Zhen-Qiu Hou, Zeng-Guang |
| author_facet | Li, Hao Zhou, Xiao-Hu Li, Shu-Hai Gui, Mei-Jiang Xie, Xiao-Liang Liu, Shi-Qi Wang, Shuang-Yi Feng, Zhen-Qiu Hou, Zeng-Guang |
| contents | Offline reinforcement learning (RL) aims to optimize a policy using collected data without online interactions. Model-based approaches are particularly appealing for addressing offline RL challenges because of their capability to mitigate the limitations of data coverage through data generation using models. Nonetheless, a prevalent issue in offline RL is the overestimation caused by distribution shift. This study proposes a novel model-based offline RL algorithm named Conservative Reward for model-based Offline Policy optimization (CROP). CROP introduces a streamlined objective that concurrently minimizes estimation error and the rewards of random actions, thereby yielding a robustly conservative reward estimator. Theoretical analysis shows that the designed conservative reward mechanism leads to a conservative policy evaluation and mitigates distribution shift. Experiments showcase that with the simple modification to reward estimation, CROP can conservatively estimate the reward and achieve competitive performance with existing methods. The source code will be available after acceptance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_17245 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | CROP: Conservative Reward for Model-based Offline Policy Optimization Li, Hao Zhou, Xiao-Hu Li, Shu-Hai Gui, Mei-Jiang Xie, Xiao-Liang Liu, Shi-Qi Wang, Shuang-Yi Feng, Zhen-Qiu Hou, Zeng-Guang Machine Learning Artificial Intelligence Offline reinforcement learning (RL) aims to optimize a policy using collected data without online interactions. Model-based approaches are particularly appealing for addressing offline RL challenges because of their capability to mitigate the limitations of data coverage through data generation using models. Nonetheless, a prevalent issue in offline RL is the overestimation caused by distribution shift. This study proposes a novel model-based offline RL algorithm named Conservative Reward for model-based Offline Policy optimization (CROP). CROP introduces a streamlined objective that concurrently minimizes estimation error and the rewards of random actions, thereby yielding a robustly conservative reward estimator. Theoretical analysis shows that the designed conservative reward mechanism leads to a conservative policy evaluation and mitigates distribution shift. Experiments showcase that with the simple modification to reward estimation, CROP can conservatively estimate the reward and achieve competitive performance with existing methods. The source code will be available after acceptance. |
| title | CROP: Conservative Reward for Model-based Offline Policy Optimization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.17245 |