Recursive Reward Aggregation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911138024783872 |
|---|---|
| author | Tang, Yuting Zhang, Yivan Ackermann, Johannes Zhang, Yu-Jie Nishimori, Soichiro Sugiyama, Masashi |
| author_facet | Tang, Yuting Zhang, Yivan Ackermann, Johannes Zhang, Yu-Jie Nishimori, Soichiro Sugiyama, Masashi |
| contents | In reinforcement learning (RL), aligning agent behavior with specific objectives typically requires careful design of the reward function, which can be challenging when the desired objectives are complex. In this work, we propose an alternative approach for flexible behavior alignment that eliminates the need to modify the reward function by selecting appropriate reward aggregation functions. By introducing an algebraic perspective on Markov decision processes (MDPs), we show that the Bellman equations naturally emerge from the recursive generation and aggregation of rewards, allowing for the generalization of the standard discounted sum to other recursive aggregations, such as discounted max and Sharpe ratio. Our approach applies to both deterministic and stochastic settings and integrates seamlessly with value-based and actor-critic algorithms. Experimental results demonstrate that our approach effectively optimizes diverse objectives, highlighting its versatility and potential for real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_08537 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Recursive Reward Aggregation Tang, Yuting Zhang, Yivan Ackermann, Johannes Zhang, Yu-Jie Nishimori, Soichiro Sugiyama, Masashi Machine Learning Category Theory In reinforcement learning (RL), aligning agent behavior with specific objectives typically requires careful design of the reward function, which can be challenging when the desired objectives are complex. In this work, we propose an alternative approach for flexible behavior alignment that eliminates the need to modify the reward function by selecting appropriate reward aggregation functions. By introducing an algebraic perspective on Markov decision processes (MDPs), we show that the Bellman equations naturally emerge from the recursive generation and aggregation of rewards, allowing for the generalization of the standard discounted sum to other recursive aggregations, such as discounted max and Sharpe ratio. Our approach applies to both deterministic and stochastic settings and integrates seamlessly with value-based and actor-critic algorithms. Experimental results demonstrate that our approach effectively optimizes diverse objectives, highlighting its versatility and potential for real-world applications. |
| title | Recursive Reward Aggregation |
| topic | Machine Learning Category Theory |
| url | https://arxiv.org/abs/2507.08537 |