A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911717604196352 |
|---|---|
| author | Shtossel, Erel Vidler, Alicia Shaham, Uri Kaminka, Gal A. |
| author_facet | Shtossel, Erel Vidler, Alicia Shaham, Uri Kaminka, Gal A. |
| contents | Recent research has revived and amplified interest in algorithms for undiscounted average reward reinforcement learning in infinite-horizon, non-episodic (continuing) tasks. Semi-Markov decision processes (SMDPs) are of particular interest. In SMDPs, discrete actions stochastically generate both rewards and durations, and the objective is to optimize the average reward rate. Existing algorithms approach this by optimizing the ratio of rewards to durations. However, when rewards and durations are non-stationary (in the infinite horizon), this can be incorrect. This paper presents a novel modified harmonic mean operator that correctly computes reward rates even under such conditions. This yields model-free learning algorithms that can work with SMDPs, while maintaining robustness to non-stationary reward and duration distributions over time. We prove theoretical properties of the modified harmonic mean operator, and empirically demonstrate its efficacy in comparison to existing algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_04880 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPs Shtossel, Erel Vidler, Alicia Shaham, Uri Kaminka, Gal A. Machine Learning Artificial Intelligence Recent research has revived and amplified interest in algorithms for undiscounted average reward reinforcement learning in infinite-horizon, non-episodic (continuing) tasks. Semi-Markov decision processes (SMDPs) are of particular interest. In SMDPs, discrete actions stochastically generate both rewards and durations, and the objective is to optimize the average reward rate. Existing algorithms approach this by optimizing the ratio of rewards to durations. However, when rewards and durations are non-stationary (in the infinite horizon), this can be incorrect. This paper presents a novel modified harmonic mean operator that correctly computes reward rates even under such conditions. This yields model-free learning algorithms that can work with SMDPs, while maintaining robustness to non-stationary reward and duration distributions over time. We prove theoretical properties of the modified harmonic mean operator, and empirically demonstrate its efficacy in comparison to existing algorithms. |
| title | A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.04880 |