Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approach
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914664033550336 |
|---|---|
| author | Tang, Dengwang Jain, Rahul Hao, Botao Wen, Zheng |
| author_facet | Tang, Dengwang Jain, Rahul Hao, Botao Wen, Zheng |
| contents | In this paper, we study the problem of efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with. We assume that the offline dataset is generated by an expert but with unknown level of competence, i.e., it is not perfect and not necessarily using the optimal policy. We show that if the learning agent models the behavioral policy (parameterized by a competence parameter) used by the expert, it can do substantially better in terms of minimizing cumulative regret, than if it doesn't do that. We establish an upper bound on regret of the exact informed PSRL algorithm that scales as $\tilde{O}(\sqrt{T})$. This requires a novel prior-dependent regret analysis of Bayesian online learning algorithms for the infinite horizon setting. We then propose the Informed RLSVI algorithm to efficiently approximate the iPSRL algorithm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_11531 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approach Tang, Dengwang Jain, Rahul Hao, Botao Wen, Zheng Machine Learning Artificial Intelligence Systems and Control 93E35 In this paper, we study the problem of efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with. We assume that the offline dataset is generated by an expert but with unknown level of competence, i.e., it is not perfect and not necessarily using the optimal policy. We show that if the learning agent models the behavioral policy (parameterized by a competence parameter) used by the expert, it can do substantially better in terms of minimizing cumulative regret, than if it doesn't do that. We establish an upper bound on regret of the exact informed PSRL algorithm that scales as $\tilde{O}(\sqrt{T})$. This requires a novel prior-dependent regret analysis of Bayesian online learning algorithms for the infinite horizon setting. We then propose the Informed RLSVI algorithm to efficiently approximate the iPSRL algorithm. |
| title | Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approach |
| topic | Machine Learning Artificial Intelligence Systems and Control 93E35 |
| url | https://arxiv.org/abs/2310.11531 |