Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.14075 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917873451008000 |
|---|---|
| author | Sun, Shuo Qi, Meng Shen, Zuo-Jun Max |
| author_facet | Sun, Shuo Qi, Meng Shen, Zuo-Jun Max |
| contents | In this work, we consider an online robust Markov Decision Process (MDP) where we have the information of finitely many prototypes of the underlying transition kernel. We consider an adaptively updated ambiguity set of the prototypes and propose an algorithm that efficiently identifies the true underlying transition kernel while guaranteeing the performance of the corresponding robust policy. To be more specific, we provide a sublinear regret of the subsequent optimal robust policy. We also provide an early stopping mechanism and a worst-case performance bound of the value function. In numerical experiments, we demonstrate that our method outperforms existing approaches, particularly in the early stage with limited data. This work contributes to robust MDPs by considering possible prior information about the underlying transition probability and online learning, offering both theoretical insights and practical algorithms for improved decision-making under uncertainty. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14075 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Online MDP with Transition Prototypes: A Robust Adaptive Approach Sun, Shuo Qi, Meng Shen, Zuo-Jun Max Machine Learning In this work, we consider an online robust Markov Decision Process (MDP) where we have the information of finitely many prototypes of the underlying transition kernel. We consider an adaptively updated ambiguity set of the prototypes and propose an algorithm that efficiently identifies the true underlying transition kernel while guaranteeing the performance of the corresponding robust policy. To be more specific, we provide a sublinear regret of the subsequent optimal robust policy. We also provide an early stopping mechanism and a worst-case performance bound of the value function. In numerical experiments, we demonstrate that our method outperforms existing approaches, particularly in the early stage with limited data. This work contributes to robust MDPs by considering possible prior information about the underlying transition probability and online learning, offering both theoretical insights and practical algorithms for improved decision-making under uncertainty. |
| title | Online MDP with Transition Prototypes: A Robust Adaptive Approach |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2412.14075 |