Risk-Averse Total-Reward Reinforcement Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917038235058176 |
|---|---|
| author | Su, Xihong Hau, Jia Lin Doko, Gersi Panaganti, Kishan Petrik, Marek |
| author_facet | Su, Xihong Hau, Jia Lin Doko, Gersi Panaganti, Kishan Petrik, Marek |
| contents | Risk-averse total-reward Markov Decision Processes (MDPs) offer a promising framework for modeling and solving undiscounted infinite-horizon objectives. Existing model-based algorithms for risk measures like the entropic risk measure (ERM) and entropic value-at-risk (EVaR) are effective in small problems, but require full access to transition probabilities. We propose a Q-learning algorithm to compute the optimal stationary policy for total-reward ERM and EVaR objectives with strong convergence and performance guarantees. The algorithm and its optimality are made possible by ERM's dynamic consistency and elicitability. Our numerical results on tabular domains demonstrate quick and reliable convergence of the proposed Q-learning algorithm to the optimal risk-averse value function. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21683 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Risk-Averse Total-Reward Reinforcement Learning Su, Xihong Hau, Jia Lin Doko, Gersi Panaganti, Kishan Petrik, Marek Machine Learning Risk-averse total-reward Markov Decision Processes (MDPs) offer a promising framework for modeling and solving undiscounted infinite-horizon objectives. Existing model-based algorithms for risk measures like the entropic risk measure (ERM) and entropic value-at-risk (EVaR) are effective in small problems, but require full access to transition probabilities. We propose a Q-learning algorithm to compute the optimal stationary policy for total-reward ERM and EVaR objectives with strong convergence and performance guarantees. The algorithm and its optimality are made possible by ERM's dynamic consistency and elicitability. Our numerical results on tabular domains demonstrate quick and reliable convergence of the proposed Q-learning algorithm to the optimal risk-averse value function. |
| title | Risk-Averse Total-Reward Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.21683 |