Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913541240389632 |
|---|---|
| author | Turcato, Niccolò Sinigaglia, Alberto Libera, Alberto Dalla Carli, Ruggero Susto, Gian Antonio |
| author_facet | Turcato, Niccolò Sinigaglia, Alberto Libera, Alberto Dalla Carli, Ruggero Susto, Gian Antonio |
| contents | Continuous control Deep Reinforcement Learning (RL) approaches are known to suffer from estimation biases, leading to suboptimal policies. This paper introduces innovative methods in RL, focusing on addressing and exploiting estimation biases in Actor-Critic methods for continuous control tasks, using Deep Double Q-Learning. We design a Bias Exploiting (BE) mechanism to dynamically select the most advantageous estimation bias during training of the RL agent. Most State-of-the-art Deep RL algorithms can be equipped with the BE mechanism, without hindering performance or computational complexity. Our extensive experiments across various continuous control tasks demonstrate the effectiveness of our approaches. We show that RL algorithms equipped with this method can match or surpass their counterparts, particularly in environments where estimation biases significantly impact learning. The results underline the importance of bias exploitation in improving policy learning in RL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_09078 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks Turcato, Niccolò Sinigaglia, Alberto Libera, Alberto Dalla Carli, Ruggero Susto, Gian Antonio Machine Learning Artificial Intelligence Continuous control Deep Reinforcement Learning (RL) approaches are known to suffer from estimation biases, leading to suboptimal policies. This paper introduces innovative methods in RL, focusing on addressing and exploiting estimation biases in Actor-Critic methods for continuous control tasks, using Deep Double Q-Learning. We design a Bias Exploiting (BE) mechanism to dynamically select the most advantageous estimation bias during training of the RL agent. Most State-of-the-art Deep RL algorithms can be equipped with the BE mechanism, without hindering performance or computational complexity. Our extensive experiments across various continuous control tasks demonstrate the effectiveness of our approaches. We show that RL algorithms equipped with this method can match or surpass their counterparts, particularly in environments where estimation biases significantly impact learning. The results underline the importance of bias exploitation in improving policy learning in RL. |
| title | Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2402.09078 |