Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909833466216448 |
|---|---|
| author | Jain, Ayush Kosaka, Norio Li, Xinhu Kim, Kyung-Min Bıyık, Erdem Lim, Joseph J. |
| author_facet | Jain, Ayush Kosaka, Norio Li, Xinhu Kim, Kyung-Min Bıyık, Erdem Lim, Joseph J. |
| contents | In reinforcement learning, off-policy actor-critic methods like DDPG and TD3 use deterministic policy gradients: the Q-function is learned from environment data, while the actor maximizes it via gradient ascent. We observe that in complex tasks such as dexterous manipulation and restricted locomotion with mobility constraints, the Q-function exhibits many local optima, making gradient ascent prone to getting stuck. To address this, we introduce SAVO, an actor architecture that (i) generates multiple action proposals and selects the one with the highest Q-value, and (ii) approximates the Q-function repeatedly by truncating poor local optima to guide gradient ascent more effectively. We evaluate tasks such as restricted locomotion, dexterous manipulation, and large discrete-action space recommender systems and show that our actor finds optimal actions more frequently and outperforms alternate actor architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11833 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions Jain, Ayush Kosaka, Norio Li, Xinhu Kim, Kyung-Min Bıyık, Erdem Lim, Joseph J. Machine Learning Artificial Intelligence Robotics In reinforcement learning, off-policy actor-critic methods like DDPG and TD3 use deterministic policy gradients: the Q-function is learned from environment data, while the actor maximizes it via gradient ascent. We observe that in complex tasks such as dexterous manipulation and restricted locomotion with mobility constraints, the Q-function exhibits many local optima, making gradient ascent prone to getting stuck. To address this, we introduce SAVO, an actor architecture that (i) generates multiple action proposals and selects the one with the highest Q-value, and (ii) approximates the Q-function repeatedly by truncating poor local optima to guide gradient ascent more effectively. We evaluate tasks such as restricted locomotion, dexterous manipulation, and large discrete-action space recommender systems and show that our actor finds optimal actions more frequently and outperforms alternate actor architectures. |
| title | Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions |
| topic | Machine Learning Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2410.11833 |