Risk-Averse Reinforcement Learning with Itakura-Saito Loss
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909622983458816 |
|---|---|
| author | Udovichenko, Igor Croissant, Olivier Toleutaeva, Anita Burnaev, Evgeny Korotin, Alexander |
| author_facet | Udovichenko, Igor Croissant, Olivier Toleutaeva, Anita Burnaev, Evgeny Korotin, Alexander |
| contents | Risk-averse reinforcement learning finds application in various high-stakes fields. Unlike classical reinforcement learning, which aims to maximize expected returns, risk-averse agents choose policies that minimize risk, occasionally sacrificing expected value. These preferences can be framed through utility theory. We focus on the specific case of the exponential utility function, where one can derive the Bellman equations and employ various reinforcement learning algorithms with few modifications. To address this, we introduce to the broad machine learning community a numerically stable and mathematically sound loss function based on the Itakura-Saito divergence for learning state-value and action-value functions. We evaluate the Itakura-Saito loss function against established alternatives, both theoretically and empirically. In the experimental section, we explore multiple scenarios, some with known analytical solutions, and show that the considered loss function outperforms the alternatives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16925 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Risk-Averse Reinforcement Learning with Itakura-Saito Loss Udovichenko, Igor Croissant, Olivier Toleutaeva, Anita Burnaev, Evgeny Korotin, Alexander Machine Learning Risk-averse reinforcement learning finds application in various high-stakes fields. Unlike classical reinforcement learning, which aims to maximize expected returns, risk-averse agents choose policies that minimize risk, occasionally sacrificing expected value. These preferences can be framed through utility theory. We focus on the specific case of the exponential utility function, where one can derive the Bellman equations and employ various reinforcement learning algorithms with few modifications. To address this, we introduce to the broad machine learning community a numerically stable and mathematically sound loss function based on the Itakura-Saito divergence for learning state-value and action-value functions. We evaluate the Itakura-Saito loss function against established alternatives, both theoretically and empirically. In the experimental section, we explore multiple scenarios, some with known analytical solutions, and show that the considered loss function outperforms the alternatives. |
| title | Risk-Averse Reinforcement Learning with Itakura-Saito Loss |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.16925 |