Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913569075888128 |
|---|---|
| author | Hau, Jia Lin Delage, Erick Derman, Esther Ghavamzadeh, Mohammad Petrik, Marek |
| author_facet | Hau, Jia Lin Delage, Erick Derman, Esther Ghavamzadeh, Mohammad Petrik, Marek |
| contents | In Markov decision processes (MDPs), quantile risk measures such as Value-at-Risk are a standard metric for modeling RL agents' preferences for certain outcomes. This paper proposes a new Q-learning algorithm for quantile optimization in MDPs with strong convergence and performance guarantees. The algorithm leverages a new, simple dynamic program (DP) decomposition for quantile MDPs. Compared with prior work, our DP decomposition requires neither known transition probabilities nor solving complex saddle point equations and serves as a suitable foundation for other model-free RL algorithms. Our numerical results in tabular domains show that our Q-learning algorithm converges to its DP variant and outperforms earlier algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_24128 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis Hau, Jia Lin Delage, Erick Derman, Esther Ghavamzadeh, Mohammad Petrik, Marek Machine Learning In Markov decision processes (MDPs), quantile risk measures such as Value-at-Risk are a standard metric for modeling RL agents' preferences for certain outcomes. This paper proposes a new Q-learning algorithm for quantile optimization in MDPs with strong convergence and performance guarantees. The algorithm leverages a new, simple dynamic program (DP) decomposition for quantile MDPs. Compared with prior work, our DP decomposition requires neither known transition probabilities nor solving complex saddle point equations and serves as a suitable foundation for other model-free RL algorithms. Our numerical results in tabular domains show that our Q-learning algorithm converges to its DP variant and outperforms earlier algorithms. |
| title | Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.24128 |