Bootstrapped Reward Shaping
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908465104945152 |
|---|---|
| author | Adamczyk, Jacob Makarenko, Volodymyr Tiomkin, Stas Kulkarni, Rahul V. |
| author_facet | Adamczyk, Jacob Makarenko, Volodymyr Tiomkin, Stas Kulkarni, Rahul V. |
| contents | In reinforcement learning, especially in sparse-reward domains, many environment steps are required to observe reward information. In order to increase the frequency of such observations, "potential-based reward shaping" (PBRS) has been proposed as a method of providing a more dense reward signal while leaving the optimal policy invariant. However, the required "potential function" must be carefully designed with task-dependent knowledge to not deter training performance. In this work, we propose a "bootstrapped" method of reward shaping, termed BSRS, in which the agent's current estimate of the state-value function acts as the potential function for PBRS. We provide convergence proofs for the tabular setting, give insights into training dynamics for deep RL, and show that the proposed method improves training speed in the Atari suite. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00989 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bootstrapped Reward Shaping Adamczyk, Jacob Makarenko, Volodymyr Tiomkin, Stas Kulkarni, Rahul V. Machine Learning Artificial Intelligence In reinforcement learning, especially in sparse-reward domains, many environment steps are required to observe reward information. In order to increase the frequency of such observations, "potential-based reward shaping" (PBRS) has been proposed as a method of providing a more dense reward signal while leaving the optimal policy invariant. However, the required "potential function" must be carefully designed with task-dependent knowledge to not deter training performance. In this work, we propose a "bootstrapped" method of reward shaping, termed BSRS, in which the agent's current estimate of the state-value function acts as the potential function for PBRS. We provide convergence proofs for the tabular setting, give insights into training dynamics for deep RL, and show that the proposed method improves training speed in the Atari suite. |
| title | Bootstrapped Reward Shaping |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2501.00989 |