Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916230223364096 |
|---|---|
| author | Madhow, Sunil Qiao, Dan Yin, Ming Wang, Yu-Xiang |
| author_facet | Madhow, Sunil Qiao, Dan Yin, Ming Wang, Yu-Xiang |
| contents | Developing theoretical guarantees on the sample complexity of offline RL methods is an important step towards making data-hungry RL algorithms practically viable. Currently, most results hinge on unrealistic assumptions about the data distribution -- namely that it comprises a set of i.i.d. trajectories collected by a single logging policy. We consider a more general setting where the dataset may have been gathered adaptively. We develop theory for the TMIS Offline Policy Evaluation (OPE) estimator in this generalized setting for tabular MDPs, deriving high-probability, instance-dependent bounds on its estimation error. We also recover minimax-optimal offline learning in the adaptive setting. Finally, we conduct simulations to empirically analyze the behavior of these estimators under adaptive and non-adaptive regimes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_14063 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data Madhow, Sunil Qiao, Dan Yin, Ming Wang, Yu-Xiang Machine Learning Artificial Intelligence Developing theoretical guarantees on the sample complexity of offline RL methods is an important step towards making data-hungry RL algorithms practically viable. Currently, most results hinge on unrealistic assumptions about the data distribution -- namely that it comprises a set of i.i.d. trajectories collected by a single logging policy. We consider a more general setting where the dataset may have been gathered adaptively. We develop theory for the TMIS Offline Policy Evaluation (OPE) estimator in this generalized setting for tabular MDPs, deriving high-probability, instance-dependent bounds on its estimation error. We also recover minimax-optimal offline learning in the adaptive setting. Finally, we conduct simulations to empirically analyze the behavior of these estimators under adaptive and non-adaptive regimes. |
| title | Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2306.14063 |