Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910967572463616 |
|---|---|
| author | Wang, Tao Zhang, Ruipeng Gao, Sicun |
| author_facet | Wang, Tao Zhang, Ruipeng Gao, Sicun |
| contents | Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL tasks. Questioning the common belief that enforcing approximate trust regions leads to steady policy improvement in practice, we show that the more critical factor is the enhanced value estimation accuracy from more value update steps in each iteration. To demonstrate, we show that by simply increasing the number of value update steps per iteration, vanilla policy gradient itself can achieve performance comparable to or better than PPO in all the standard continuous control benchmark environments. Importantly, this simple change to vanilla policy gradient is significantly more robust to hyperparameter choices, opening up the possibility that RL algorithms may still become more effective and easier to use. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19247 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving Value Estimation Critically Enhances Vanilla Policy Gradient Wang, Tao Zhang, Ruipeng Gao, Sicun Machine Learning Artificial Intelligence Robotics I.2.6 Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL tasks. Questioning the common belief that enforcing approximate trust regions leads to steady policy improvement in practice, we show that the more critical factor is the enhanced value estimation accuracy from more value update steps in each iteration. To demonstrate, we show that by simply increasing the number of value update steps per iteration, vanilla policy gradient itself can achieve performance comparable to or better than PPO in all the standard continuous control benchmark environments. Importantly, this simple change to vanilla policy gradient is significantly more robust to hyperparameter choices, opening up the possibility that RL algorithms may still become more effective and easier to use. |
| title | Improving Value Estimation Critically Enhances Vanilla Policy Gradient |
| topic | Machine Learning Artificial Intelligence Robotics I.2.6 |
| url | https://arxiv.org/abs/2505.19247 |