Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909333957115904 |
|---|---|
| author | Chaudhari, Shreyas Deshpande, Ameet da Silva, Bruno Castro Thomas, Philip S. |
| author_facet | Chaudhari, Shreyas Deshpande, Ameet da Silva, Bruno Castro Thomas, Philip S. |
| contents | Evaluating policies using off-policy data is crucial for applying reinforcement learning to real-world problems such as healthcare and autonomous driving. Previous methods for off-policy evaluation (OPE) generally suffer from high variance or irreducible bias, leading to unacceptably high prediction errors. In this work, we introduce STAR, a framework for OPE that encompasses a broad range of estimators -- which include existing OPE methods as special cases -- that achieve lower mean squared prediction errors. STAR leverages state abstraction to distill complex, potentially continuous problems into compact, discrete models which we call abstract reward processes (ARPs). Predictions from ARPs estimated from off-policy data are provably consistent (asymptotically correct). Rather than proposing a specific estimator, we present a new framework for OPE and empirically demonstrate that estimators within STAR outperform existing methods. The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02172 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation Chaudhari, Shreyas Deshpande, Ameet da Silva, Bruno Castro Thomas, Philip S. Machine Learning Artificial Intelligence Evaluating policies using off-policy data is crucial for applying reinforcement learning to real-world problems such as healthcare and autonomous driving. Previous methods for off-policy evaluation (OPE) generally suffer from high variance or irreducible bias, leading to unacceptably high prediction errors. In this work, we introduce STAR, a framework for OPE that encompasses a broad range of estimators -- which include existing OPE methods as special cases -- that achieve lower mean squared prediction errors. STAR leverages state abstraction to distill complex, potentially continuous problems into compact, discrete models which we call abstract reward processes (ARPs). Predictions from ARPs estimated from off-policy data are provably consistent (asymptotically correct). Rather than proposing a specific estimator, we present a new framework for OPE and empirically demonstrate that estimators within STAR outperform existing methods. The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases. |
| title | Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.02172 |