Robust $Q$-learning Algorithm for Markov Decision Processes under Wasserstein Uncertainty
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911925279916032 |
|---|---|
| author | Neufeld, Ariel Sester, Julian |
| author_facet | Neufeld, Ariel Sester, Julian |
| contents | We present a novel $Q$-learning algorithm tailored to solve distributionally robust Markov decision problems where the corresponding ambiguity set of transition probabilities for the underlying Markov decision process is a Wasserstein ball around a (possibly estimated) reference measure. We prove convergence of the presented algorithm and provide several examples also using real data to illustrate both the tractability of our algorithm as well as the benefits of considering distributional robustness when solving stochastic optimal control problems, in particular when the estimated distributions turn out to be misspecified in practice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2210_00898 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Robust $Q$-learning Algorithm for Markov Decision Processes under Wasserstein Uncertainty Neufeld, Ariel Sester, Julian Machine Learning Artificial Intelligence Optimization and Control Probability We present a novel $Q$-learning algorithm tailored to solve distributionally robust Markov decision problems where the corresponding ambiguity set of transition probabilities for the underlying Markov decision process is a Wasserstein ball around a (possibly estimated) reference measure. We prove convergence of the presented algorithm and provide several examples also using real data to illustrate both the tractability of our algorithm as well as the benefits of considering distributional robustness when solving stochastic optimal control problems, in particular when the estimated distributions turn out to be misspecified in practice. |
| title | Robust $Q$-learning Algorithm for Markov Decision Processes under Wasserstein Uncertainty |
| topic | Machine Learning Artificial Intelligence Optimization and Control Probability |
| url | https://arxiv.org/abs/2210.00898 |