Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913301539061760 |
|---|---|
| author | Seyde, Tim Werner, Peter Schwarting, Wilko Wulfmeier, Markus Rus, Daniela |
| author_facet | Seyde, Tim Werner, Peter Schwarting, Wilko Wulfmeier, Markus Rus, Daniela |
| contents | Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer in the absence of action penalization in line with optimal control theory. In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency, but action costs can be detrimental to exploration during early training. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution, taking advantage of recent results in decoupled Q-learning to scale our approach to high-dimensional action spaces up to dim(A) = 38. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04253 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution Seyde, Tim Werner, Peter Schwarting, Wilko Wulfmeier, Markus Rus, Daniela Machine Learning Artificial Intelligence Robotics Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer in the absence of action penalization in line with optimal control theory. In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency, but action costs can be detrimental to exploration during early training. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution, taking advantage of recent results in decoupled Q-learning to scale our approach to high-dimensional action spaces up to dim(A) = 38. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks. |
| title | Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution |
| topic | Machine Learning Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2404.04253 |