Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910273110016000 |
|---|---|
| author | Nishimori, Soichiro Parmas, Paavo Koyamada, Sotetsu Kozuno, Tadashi Kitamura, Toshinori Ishii, Shin Matsuo, Yutaka |
| author_facet | Nishimori, Soichiro Parmas, Paavo Koyamada, Sotetsu Kozuno, Tadashi Kitamura, Toshinori Ishii, Shin Matsuo, Yutaka |
| contents | In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce uncertainty; without such retries, a greedy policy is optimal. We formalize this intuition with ReMax, an objective that evaluates a policy by the expected maximum return over $M$ samples, where $M$ is a positive integer, while accounting for return uncertainty. Optimizing this objective induces stochastic exploration as an emergent property, without explicit bonus terms. For efficient policy optimization, we derive a new policy-gradient formulation for ReMax and introduce ReMax PPO (RePPO), a PPO variant that optimizes ReMax while generalizing the discrete retry count $M$ to a continuous parameter $m > 0$, enabling fine-grained control of exploration. Empirically, RePPO promotes exploration, without any explicit exploration bonuses, on the MinAtar and Craftax benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_00151 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying Nishimori, Soichiro Parmas, Paavo Koyamada, Sotetsu Kozuno, Tadashi Kitamura, Toshinori Ishii, Shin Matsuo, Yutaka Machine Learning Artificial Intelligence In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce uncertainty; without such retries, a greedy policy is optimal. We formalize this intuition with ReMax, an objective that evaluates a policy by the expected maximum return over $M$ samples, where $M$ is a positive integer, while accounting for return uncertainty. Optimizing this objective induces stochastic exploration as an emergent property, without explicit bonus terms. For efficient policy optimization, we derive a new policy-gradient formulation for ReMax and introduce ReMax PPO (RePPO), a PPO variant that optimizes ReMax while generalizing the discrete retry count $M$ to a continuous parameter $m > 0$, enabling fine-grained control of exploration. Empirically, RePPO promotes exploration, without any explicit exploration bonuses, on the MinAtar and Craftax benchmarks. |
| title | Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2606.00151 |