MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909172228947968 |
|---|---|
| author | Liu, Xiao-Yin Zhou, Xiao-Hu Li, Guotao Li, Hao Gui, Mei-Jiang Xiang, Tian-Yu Huang, De-Xing Hou, Zeng-Guang |
| author_facet | Liu, Xiao-Yin Zhou, Xiao-Hu Li, Guotao Li, Hao Gui, Mei-Jiang Xiang, Tian-Yu Huang, De-Xing Hou, Zeng-Guang |
| contents | Offline reinforcement learning (RL) faces a significant challenge of distribution shift. Model-free offline RL penalizes the Q value for out-of-distribution (OOD) data or constrains the policy closed to the behavior policy to tackle this problem, but this inhibits the exploration of the OOD region. Model-based offline RL, which uses the trained environment model to generate more OOD data and performs conservative policy optimization within that model, has become an effective method for this problem. However, the current model-based algorithms rarely consider agent robustness when incorporating conservatism into policy. Therefore, the new model-based offline algorithm with a conservative Bellman operator (MICRO) is proposed. This method trades off performance and robustness via introducing the robust Bellman operator into the algorithm. Compared with previous model-based algorithms with robust adversarial models, MICRO can significantly reduce the computation cost by only choosing the minimal Q value in the state uncertainty set. Extensive experiments demonstrate that MICRO outperforms prior RL algorithms in offline RL benchmark and is considerably robust to adversarial perturbations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_03991 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator Liu, Xiao-Yin Zhou, Xiao-Hu Li, Guotao Li, Hao Gui, Mei-Jiang Xiang, Tian-Yu Huang, De-Xing Hou, Zeng-Guang Machine Learning Artificial Intelligence Offline reinforcement learning (RL) faces a significant challenge of distribution shift. Model-free offline RL penalizes the Q value for out-of-distribution (OOD) data or constrains the policy closed to the behavior policy to tackle this problem, but this inhibits the exploration of the OOD region. Model-based offline RL, which uses the trained environment model to generate more OOD data and performs conservative policy optimization within that model, has become an effective method for this problem. However, the current model-based algorithms rarely consider agent robustness when incorporating conservatism into policy. Therefore, the new model-based offline algorithm with a conservative Bellman operator (MICRO) is proposed. This method trades off performance and robustness via introducing the robust Bellman operator into the algorithm. Compared with previous model-based algorithms with robust adversarial models, MICRO can significantly reduce the computation cost by only choosing the minimal Q value in the state uncertainty set. Extensive experiments demonstrate that MICRO outperforms prior RL algorithms in offline RL benchmark and is considerably robust to adversarial perturbations. |
| title | MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2312.03991 |